Walter Almeida's Blog

Great resource for SilverLight development on SharePoint 2010

Here is a great resource about SilverLight development on SharePoint 2010:

http://code.msdn.microsoft.com/SP2010Silverlight/

Include sample code and a video explaining how to develop with SilverLight for SharePoint 2010. Including topics like: SilverLight WebPart development, SilverLight field controls development, the use of the HTML bridge to pass data between SilverLight WebPart and the web page. And last but not least: how to connect SilverLight WebParts with no page refreshes and using the standard WebPart connection mechanism.

Also a sample making use of the new restfull access to SharePoint lists using ADO.NET services.

A must read for those who are starting developping with SilverLight on SharePoint 2010!

The case for Model Driven Software Development (Part 2) Developing high quality, well designed software applications is hard and expensive – MDSD can help!

In the first part of this series, I gave a quick introduction of the MDSD approach. Let's now enter in more details. I will explain here one of the main motivations of going for the model driven paradigm shift or: how a model driven approach can dramatically help make developing high quality software faster and easier. 

Part 1: Toward generative programming? The case for Model Driven Software Development: A quick overview of the MDSD approach

Part 2: The case for Model Driven Software Development - Developing high quality, well designed software applications is hard and expensive – MDSD can help!

Developing high quality, well designed software applications is hard and expensive 

The benefits of designing and implementing applications using a proper layered architecture approach, good separation of concerns, relevant design patterns to make the solution flexible, extensible and maintainable are not to be proven anymore and is a common understanding in the industry. However designing and implementing high quality software this way does take time and is expensive. And this will NOT help you deliver faster. YES you will produce better quality software but it will not necessarily make you more productive, at least not in the short term. It does not mean you should develop bad quality software and YES developing quality software with good design practices will in the medium/long term range be VERY profitable and even open windows for extra evolutions you did not even thought of up-front. But it can prove very difficult to sell quality practices for short projects where the extra cost and time involved for doing a state of the art design up-front will be overkill. And not only the design but also the implementation phase on a highly engineered application can cost you extra resources and time. If you indeed have a layered architecture, then implementing a new functionality will most probably involve writing code in all layers. And if you have designed your application using design patterns, then you will probably have to regularly create a set of new classes for any new feature in your application. A good example is the MVP pattern: for any new user interface you want to include, you have to create not one but at least three classes for that matter.

   

Maintaining a software application is expensive

Many studies state that the TCO (Total Cost of ownership) of a software application for a big part due to maintenance and evolution AFTER deployment to production! This means that what is expensive is not so much the development of an application but its exploitation and maintenance over time.

Implementing a well-designed, state-of-the-art application requires validation and any new development should go through a full validation process, may it be manual (peer reviews), automatic, or both. Don't do that and you can be sure that your quality will degrade rather quickly with time, and inconsistency and hacks will be introduced in your design. And the more engineered your software application is, the more difficult will it be to keep it this way and not introduce bad practices or hacks in the design, which could prove disastrous in the long term. Yes you can document your technical architecture and design principles but it means that it is again extra cost involved, thus increasing the TCO of your application. Moreover you need to find a way to keep the knowledge and know-how in the company and make sure that any person leaving the company does not do so before a proper knowledge transfer inside the company…

On the other hand, a poorly designed software application will most probably cost even more to maintain and evolve. And maintenance and evolution of such an application can to some extend become so difficult that no-one dares touching too much the code base, thus decreasing the ability for the application to evolve and decreasing the application users' satisfaction.

The case for the model driven approach: make quality easy and cost effective

YES it is very frustrating when a client tells you that he understands where you come from with your idea of writing state of the art, well engineered solutions, but that unfortunately for his project he just wants something delivered fast and at a reasonable price. And unfortunately this scenario is not uncommon. So what to do? Forget your dreams to create great designs and develop applications as they should be? Forget about the benefits of well designed quality software solutions for your clients with low budget or working with tough time constraints?

Well this is where proper tooling and especially the model driven approach can help in this direction. Because a properly targeted DSL and associated tools can eliminate the extra effort required to develop a highly engineered solution by automating the creation of all the classes, components and other artifacts required to full-fill the application design. Of course what requires (a lot of) time is the writing of the DSLs and associated tools that you need. However this work will then be reusable for all the features of the application, for the evolutions of the application and for future projects. This makes the initial investment very profitable, not only long-term on the initial project, but even short term for the actual project and for any new project starting. One way to overpass the extra cost involved in writing the DSLs and tools is to find on the market existing DSLs or at least existing tools that can help you quick-start the writing of your own DSLs. Finding already existing functional DSLs that suit your business is hard as any business has its own specificities and rules. However finding a technical DSL is easier as they are reusable horizontally across domains. And I will again give the example of a now very common and well known DSL: the ORM tools. But other more advanced tools exist to generate more than just the database access and persistence part of the application.

   

The case for the model driven approach: reduce the TCO by reducing maintenance costs  

One more benefit of using a DSL approach is also the following: when you are using a DSL to model and generate parts or all of your application, then evolutions of the application will be implemented by changing the model and re-generating the application. Therefore you (almost) don't have the risk to degrade your design quality by not respecting the architectural rules. And you don't need the extra validation step to make sure the evolutions are implemented respectful of the original design principals of your architecture. This is because the design rules are enforced by the DSL itself and the code generation process! Therefore a successful model driven strategy will also result in reducing the total cost of ownership by dramatically reducing the cost of maintenance and evolutions and even further: enforce the software quality in the long term.

We therefore talked about what I consider being the main motivation for adopting the model driven software development approach: the MDSD approach enables the cost effective implementation of high quality, highly engineered software applications while dramatically reducing the maintenance and evolution costs and enforcing the application quality and stability in the long term.

I will keep writing more on the model driven approach, so stay in touch if you like the topic and enjoyed the reading!

Toward generative programming? The case for Model Driven Software Development (Part 1): A quick overview of the MDSD approach.

I am starting now a series of posts on the Model Driven Software Development (MDSD) approach. The reason is that I strongly believe it is a very powerful approach and I will invest time and efforts in 2009 in this direction. There are already tools that are following this approach and probably one of the most compelling success stories is the ORM tools. There are also things going on in the industry and for instance Microsoft is planning to follow the model driven path with its OSLO platform and the M language, so I will also be talking about this.    

The first post of the series is a quick overview of the MDSD approach. The objective here is not to do a full presentation of this software development paradigm but introduce the subject, key terms and overall principles. I will in further posts analyze and foresee what benefits could bring MDSD to the software industry and what place this might have in the future.

Part 1: Toward generative programming? The case for Model Driven Software Development: A quick overview of the MDSD approach

Part 2: The case for Model Driven Software Development - Developing high quality, well designed software applications is hard and expensive – MDSD can help!

 

MDA versus MDSD   

In order to briefly explain what the model driven approach is, it is interesting to start explaining what the Model Driven Architecture (MDA) approach is and how it differs with a more permissive and flexible MDSD (Model Driven Software Development) approach.       

The MDA approach is an initiative of the Object Management Group (OMG), and is a natural evolution after the introduction of the UML language and its relatively wide adoption on the market. The MDA approach aims at providing a generic, model oriented, tools friendly language that could be used to automate the generation of software applications through the creation of platform independent models and diagrams. This generative implementation is supposed to be done through MDA tools, able to parse the platform independent MDA model and generate code and artifacts for the targeted platform.     

One primary objective of the MDA approach is the platform independence and a universal generic language to describe systems, the same way UML was designed to be generic enough to be usable to describe the different architectural views of a software system, and even potentially apply beyond software systems.     

While this generic MDA approach has the definite benefit of the universality of a language that anyone, and any tool could understand and exploit, it also has its drawbacks. The main drawback being that a too generic and universal language can be too much abstracted from the real system and its specificities. Extensions of the UML language such as stereotypes were invented to solve this problem but make the whole exercise of precisely diagramming a system very difficult to achieve. And the precision and level of details required for a generative approach is of crucial importance, much more than for creating diagrams for the sole purpose of communication and documentation.   

I personally am a big believer of more pragmatic approaches that aim at giving a clearer, less abstracted vision, but closer to the reality of the system. After all, the primary objective is to build systems that work and is not to have a generic description of systems for just the sake of purity. This is for me the reason why the MDA approach was not as successful and broadly used as expected and the driving force why a more permissive and flexible MDSD approach has a much better chance to succeed and convince the community and the market. 

From now on, we will talk about the Model Driven Software Development (MDSD) approach as a generic term that applies to any solution, tools or processes that follow the model driven concept of modeling an application for the purpose of automating its implementation. The MDA approach in this perspective is a MDSD approach that has the specificity of being tightly coupled to, and constrained by the UML language.

So the generic MDSD approach we describe here differs from the MDA approach in that there is not such a thing as a ready to use, generic MDSD approach and associated languages that you can use out of the box on your projects. The MDSD approach is more of a concept, a guidance and new way/philosophy to see software development. So where there is only one MDA standardized set of languages, there are not only one but as many different MDSD tools and processes as vendors or individuals will invest on creating.

   

Domain Specific Languages (DSL) and MDSD 

Let's now have a word on the Domain Specific Language (DSL) terminology. The DSL acronym is often misleadingly used for describing a full model driven approach. However a domain specific language is not really a model driven approach on its own but only a part of a full MDSD approach. The MDSD approach is more than just languages, it is a full set of languages (and more specifically DSLs), tools, runtime frameworks, code generators and/or language interpreters (more on compiled versus interpreted DSL later), on potentially more.    

Therefore DSLs are a major part of (and only a part of) the model driven approach. A DSL is an abstracted language that aims at describing a system or part of a system that we want to model in the most convenient and natural way for the targeted domain. A DSL can take any form as best suited for the targeted domain, can be textual, graphical or both or even take another exotic form.   

A DSL can be architecture oriented of functional oriented. An architecture oriented DSL aims at describing technical layers or architecture centric components and services of your application. A very common example of an architecture oriented DSL are the Object Relational Mapping tools (see the description of Llblgen for an example of such a tool). An architecture oriented DSL can be used horizontally across different functional domains. A functional DSL is a DSL that is explicitly dedicated for a given functional domain and an example can be FpML, used to model financial instruments. Functional DSLs provide business analysts with a proper abstraction level to describe their very specific domain that can then translate to code (the dream of having analysts or business specialists designing applications with no more need for developers...), or more realistically functional DSLs provide developers with a language that can help then translate functional models to applications and code.      

And finally DSLs can be interpreted or compiled. A model written in an interpreted DSL is used "as is" in the real production application, hosted by a runtime (interpreter) able to understand the model and execute operations. In my very basic understanding the M Language (Microsoft OSLO) can partly be used as such an interpreted DSL (more on OSLO on a later post). On the other hand, for compiled DSL, a model has to go through a compilation phase for transformation of the model to a lower level language, such as C# for the Microsoft platform. An example of such a compiled DSL again is the Llblgen ORM tool. It is pretty clear that compiled and interpreted DSL are very different in nature and imply very different constraints.   

In the model driven approach, the DSL or set of DSLs is the way to describe the model of the application you are aiming at implementing. Defining the model of your application is the first step in the model driven approach, followed by the generation activity, or interpretation of the model (more later on compiled versus interpreted DSLs).

   

Generators and Interpreters

The second brick of the model driven approach is the generator(s) and/or interpreter(s). The generators responsibility is to generate from the model anything you need for your specific domain. Artifacts that are useful to be generated range from code to configuration files though user manual, or anything you might dream of automating. Generation can be based on text templates that are useful to customize or extend the content of the generation. On the other hand, interpreters don't generate anything but rather interpret at runtime the models to deliver the required application functionality. A model driven approach can make use of a runtime framework. The final application will then use at runtime the support libraries that make up the framework. This is especially true and mandatory for the interpreted approach but also true for the generative approach. It indeed often proves useful to generate code and classes that are depending and built on top of a well designed and fully featured framework rather than generate the full application code base. This runtime support framework is the third brick of the MDSD approach.   

The fourth brick on a full MDSD approach would be the SDK that allows users of the DSL and generators/interpreters to extend or automate the DSL and generators/interpreters, from instantiation of a model to generation of the application, through integration with other DSLs. I will in a later post talk about DSLs composition and DSL integration with other approaches and tools, to bring the model driven approach even further.

    

That's enough for this quick introduction. Hope it was clear enough, and I will enter in more details in further posts. I will talk more in next posts on the motivations behind the model driven approach and why MDSD is good for you!

And here are a few links to resources related to the model driven approach:

MDA web site: http://www.omg.org/mda/

Microsoft OSLO web site: http://www.microsoft.com/soa/products/oslo.aspx

Podcasts related to the model driven approach:

Software Engineering radio shows on MDSD

Software Engineering radio shows on DSLs

  

Golden Rule #2: Don’t be afraid of refactoring!

Here I am, back after more than a month publishing the first golden rule. I said to myself today that I should keep up to speed with this subject in order to finally end-up with a good number of rules as I find the exercise interesting and opening great paths for thinking and reflecting on limited but compelling subjects.

So the Golden Rule #2 is about refactoring code and software architectures. One may wonder: "Why are you talking about continuous refactoring as a Golden rule??!! After all if you follow development best practices and apply all golden rules, aren't you then expecting that the design and code produced are of good quality, therefore not requiring refactoring?!"

Introduction - what is all the fuzz around refactoring?

Refactoring is often associated to "repairing" legacy code or poorly designed applications. Even though it is sometimes what refactoring is used for, refactoring is also first class citizen of a full agile process of developing high quality applications. I would even go one step further: if a reasonable sized project is achieved with no refactoring phase during its implementation, then there is probably something wrong and a flawed underlying design that will bring you trouble on the path ahead.

Software development is a complex discipline, and the vast number of patterns, tools, frameworks and best practices leads to numerous ways for implementing one scenario. It is all about pros and cons, compromises, and identifying the potential future points of evolutions of the business need in order to be ready for it. 

Even though it is always recommended to do a design first and coding later, I don't believe one could or should do a fully detailed design before writing the first line of code. The key winning idea is to be pragmatic, agile and iterative. Any design should be validated by proper implementation and validation, to then be adapted, enhanced and implemented again. There is a limit to paper diagrams and even experienced designers cannot fully predict the behaviour of the future implementation of there designs in terms of performance, security, usability and fitting in the target infrastructure, nor can predict that the frameworks and techniques that they theoretically plan on using will actually be relevant perform/provide a service as excepted; one exception is if they already designed and developed a very similar application, following very similar constraints and non-functional requirements, and targeted to a very similar platform. Added to these technical only aspects, business needs can also evolve, during the lifecycle of the project, get re-scoped and adapted and therefore require changes in the application technical design. The consequence of this is that refactoring phases will be mandatory to iteratively build the software solution.

 

Another point is that a successful software architecture and design is the result of a collaborative work between several people, architects, analysts, and developers. So, all participants should be involved in the process as soon as possible thus requiring starting the development phase early for efficient and productive feedback from the development side ASAP. As a consequence, refactoring phases will have to be planed and performed until reaching an acceptable quality level.

So why do you need refactoring?

Including refactoring phases in your development process and lifecycle answers to the following needs:

• You need to make sure that your software design and architecture properly address evolving business needs: Refactoring phases can be necessary if a major business evolution requires rethinking of part of the design of the application. The application should always be designed with flexibility in mind and requires that aspects of potential evolutions are properly identified to know where to design and build flexibility.

• In an agile process, refactoring is an essential technique to iteratively build the most appropriate software architecture and design, while gaining in maturity iteration by iteration. This is a key for agile development where functional slices of the application are frequently delivered and most of the times require partial refactoring to move from one iteration to the other and increase quality and relevance of the design.

• Late discovery of functional or technical constraints can lead to the need for a partial refactoring of the application to address this unexpected event. By the way, an agile, highly iterative process can help reducing the risk by early addressing all the aspects of the application, and especially the aspects that are considered high risk.

• Results of test campaigns can reveals areas that need refactoring, for either technical reasons (poor performance, low scalability, etc.) or functional reasons (inappropriate functional behaviour revealed by the UAT that has an impact on technical design). Again: and agile iterative process will help address early these issues.

Rigorously including refactoring phases in your development process will therefore help you increase the quality and relevance of the overall application design, making the application more maintainable, robust and flexible. Moreover, being able to regularly and efficiently perform refactoring sessions without breaking the whole application and entering uncontrolled regression phases will be the undisputable proof that your software design is effectively of high standard.

 

Can I include refactoring in my development process seamlessly and for free?

Ok, refactoring is not a simple thing and it is definitively not a painless, safe action to perform if you are not prepared for it.

First condition to include regular refactoring phases as part of your development process is to actually have a well designed and architected application. And refactoring becomes a way to iteratively increase the quality of this design. But you can't apply proper refactoring as part of the development process for commando like or any other kind of quick and dirty type of programming.

Second condition is that you should have a proper set of unit tests covering your full code base. Refactoring is about rewriting and restructuring code thus meaning creation of new bugs. And you don't want to break what was working before. Therefore a set of unit test with high code coverage will help you (yet not always suffice) verify the quality of the refactored code and make corrections if required. Important refactoring will also require the re-writing of the unit tests, which is ok: just make sure that your new set of unit tests still covers the same functional use cases.

Refactoring can be painful if you don't have the proper tooling to assist you and automate repetitive operations. And the pain can be even worth if you need to refactor across several projects. Visual Studio includes basic refactoring tools that are really useful and a big step forward since version 2005. But you can go even a step further and use commercial refactoring tools. The one I use (and love!) is ReSharper, and you can read my post about it for more details.

Another thing you need is a refactoring compliant source code control system that allows you to seamlessly move and rename files while keeping the integrity of source code repository and version history. And I haven't found such a tool yet. To be honest I haven't tried too many tools as I am quite happy with the Microsoft Team Foundation Server package for building a full continuous integration environment. Unfortunately, TFS does not quite like renaming and moving files and can act weird when you try to do so. For large refactoring, the solution is to start on a new, clean TFS project that is built on the new file structure implied by your refactoring. You still keep the full version history though the previous TFS project but lose the connection with your new project. Another advice to limit the issues you can have with your source control while refactoring is the following: try to avoid to a too deep hierarchy of projects, trying to mirror the namespace tree. Just save your projects on a flat, or as flat as possible, file structure. You still can have a hierarchical view in Visual Studio by creating solution folders. This way, it will dramatically make moving or reorganising namespaces much easier! Don't forget that reorganisation namespace is one of the most common thing you will do while refactoring...

Last point to mention is that it will be useful to have an automated process for updating documentation. Refactoring implies changes in the documentation and documentation is not a good candidate for manual update because it will then never be updated. Solution for the documentation is the use of tools that automatically reflect on the code comments to generate the detailed documentation. So rule is: comment properly your code and run a tool on your code base. The tool I use is Sandcastle. And for more high level class diagrams, you can choose a UML tool that can reverse code-engineer UML diagrams from you code. Visual Studio embeds one very basic tool. I personally use Enterprise Architect that I will describe in a future post. Key element is that you should be able to automatically update the technical documentation if you want to have technical documentation at all!

 

So am I ready to go?

You must be aware that refactoring cycles are time consuming, require courage and patience for the extra effort induced. It therefore requires discipline and a well accepted and applied process that includes these "technical" refactoring phases, apparently not bringing any additional features or business value, but really mandatory in the medium to long term for creating a robust and flexible applications. Therefore the sole idea of technical refactoring being part of the software application development lifecycle is something that needs to be strongly supported by the project management board, and more importantly: really applied during the project lifetime. And this is probably the most difficult part as management teams can have difficulties understanding the need of spending a lot of time, effort and money for something that apparently (but apparently only) does not bring any added value to their business application. And this difficulty can be caused by a lack of understanding of both parties and a low trust given to others.

This leads us to another Golden rule on which I will write later. This Golden rule is not really a development golden rule but a more high level, project level, strategic golden rule: go for collaborative and responsibility driven team organisation rather than for a hierarchic centric organisation. This rule says the following: to make a project successful (be it a software project or any kind of project BTW), you should build a team with well defined responsibilities and skills and trust everyone in the team for his/her competencies and believe and take into consideration the well-founded of their advice. This is the best and more human way IMHO to have a real success on your projects and long term involvement and efficiency of your overall project team. But more on this later, so keep connected to the Golden rules series!!

Is the Microsoft Entity Framework more than just an ORM?

I recently listened to the show #369 of .Net Rocks! And read the reaction of Frans Bourma (LLBLGenPro) on the subject.

I wanted to react and give my views on the subject. 

For an overview of what Object Relational Mapping is, please read my previous post on the subject.  

This post assumes that the reader will already have the knowledge of the Microsoft Entity Framework concepts. If it is not the case please read the ADO.NET Entity Framework Overview as a pre-requisite.

Quick introduction to the Microsoft Entity Data Model (EDM)   

The MS Entity Framework is composed of several parts, one of which is the Entity Data Model (EDM). The EDM is used for modelling at a conceptual level the business space, and actually corresponds to the "object space" of the Object/Relational mapping. The EDM is the model describing the set of objects mapped to relational tables on the database side. Then comes the mapping layer, being the bridge or glue between the EDM, or "object side" and the tables in your relational database. Finally, the Object Relational Mapping engine does the trick to manipulate data, persist changes to the database and feed the EDM objects at runtime. An important aspect that will be essential for the main subject of the post is the following: the EDM model is decoupled from the mapping layer and from the ORM engine. In other words the EDM model has no knowledge that there is a mapping and an ORM engine at all, and can be used or re-used without the presence of the mapping and ORM engine. This is an essential pattern for working in a remote/disconnected scenario where a service is responsible for manipulating and persisting data (and thus is using the mapping layer and ORM engine), and the client solely use the data provided by the services, through the form of EDM objects, with no knowledge on how and where the data is stored. Please note that there is nothing conceptually new here and this is the basis of most existing ORM tools of the market, adapted to the Microsoft terminology.

So now that the introduction is made, here comes the main subject of this post.

The Microsoft Entity Framework entering the ORM tools market

For what I've understood from the .Net Rocks podcast, Microsoft could be having more interest on the EDM model than on the ORM engine. You could imagine that this is because they did not manage to catch-up with the existing ORM tools maturity or for a more strategic reason, but that's out of subject here.

Fact is they are many ORM vendors, doing and mastering this job for years and being very mature. The EDM specifically could be on its side more interesting for MS because it is platform oriented and could become a new Microsoft standard, ideal for making applications inter-communicate though this EDM full object oriented, strongly typed object model. Let be clear: the Entity Framework is not a platform, however the Entity Framework and more specifically the EDM model can become an important glue element of the Microsoft platform, becoming the de-facto standard for inter-connecting Microsoft applications and services and any third party application wanting to leverage the EDM opportunity and offer a communication interface to the outer world though EDM entities.

  

A new business opportunity for third party vendors?

What I would extrapolate from Stephen Forte talk in the podcast (and that engages only me) is that there is an opportunity for ORM vendors to build on top of EDM and provide advanced and efficient ORM engine core functionalities: a third party ORM engine replacing the Microsoft one, manipulating and persisting data all through EDM entities. There is also an opportunity for Microsoft Partners to build tools or designers that could assist creating a EDM model and map it to the database + making use of an ORM engine (being the Microsoft one or any other "EDM Compliant" ORM engine). In this scenario, third parties ORM vendors would be using the EDM as the data model used to manipulate, transfer data, between applications and services. Another clarification: the EDM is indeed not a new, revolutionary concept, and every ORM tool in the market is including such a concept: the entities model is indeed the actual object oriented side of the object/relation mapping layer and is an essential part of any ORM tool.

    

Closer look on the EDM on a broader picture

The EDM model becomes a data contract in the WCF sense, making easier and efficient to exchange data between systems, regardless of where the underlying data is stored and whether or not we are using ORM engine to concretely retrieve these EDM entities. For instance you could most probably use and EDM model to represent data in an inter-application communication scenario and not link to any database in the back-end. The EDM and the Entity Framework ORM engine are loose-coupled and you can use one or the other independently. Well at least you can use the EDM independently from the EF ORM engine, I am not too sure of the opposite.

And the EDM is even more than just a data contract: it comes with tools and will probably come with more. One is the Entity SQL, based on SQL syntax oriented toward object queries on the EDM model. And again this is nothing new: for instance Llblgen provides a full framework around its entity model, adding extra functionalities as compared to a basic object model or to any model you could build from scratch by hand, in a reasonable amount of time.

So back to the point: using the EDM as a data contract, it removes the need for applying a DTO pattern for interoperability between systems and especially when exposing services and wanting to make your clients unaware and decoupled from your inner implementation and especially from the ORM tool you are using. Well to be more precise: if the EDM becomes a "Microsoft standard", any one agreeing on that could consume a service delivering EDM entities. But you should be aware that the interoperability will anyway not be complete as using EDM will probably tight you to the Microsoft platform and "Microsoft Entity Framework aware" clients. I will write another post about the DTO pattern in a SOA world and when to use or not use it. DTO is yet another mapping layer and one can wonder why after mapping from the relational world to the object world we still need another translation. My recommendation would be that one should not introduce another mapping layer is there is no absolute need for it. You might need it or might absolutely want to forget about it, depending on the scenario you're in, there is no single solution to all problems.

So is the Entity Data Model much bigger than a just an ORM?

So Yes, for what the EDM is concerned and what you can hear in the podcast as being "more than just ORM", you can achieve the same result with other ORM tool and I am actually doing it on my projects with the LLBLGen Pro generated entities model layer. What however the EDM might provide as a plus in the future is integration in the whole MS suite and become the missing glue link to seamlessly connect systems though the strongly type, object oriented EDM data model. Well let´s be more precise: the missing link in systems based on Microsoft tools and technology.   

So to insist on this point, I can understand Frans Bourma when he is reacting strongly on the sentence "The EF is much more than an ORM" because no, there is no magic and what MS say they can do with the entity framework and the EDM could be achieved with another ORM tool and its own entity model. It is just after all a matter of convention and all actors standardizing on the same framework (and for instance here the EDM part of entity framework). But Yes MS can bring something more with the EDM that goes beyond the ORM aspect, just by the fact that they can push for using the EDM as the data contract layer for all of their products. They can do it because they own the product! And this is what makes the idea powerful: for instance when a company start investing in Microsoft, it usually spread the whole SI and range of applications: start installing MS servers and desktops windows OS and you end up using Exchange server and SharePoint server and office and more and more, rather that competing products. And why? Just because all integrate so well: it is all coming for a single vendor who actually makes the integration of all its range of applications as easy and straight-forward as possible! Same thing can happen with the EDM on a more technical, low level I must say, but following the same driving idea.

      

To conclude …   

To conclude, what I can foresee is that what will bring most to Microsoft strategically speaking is the EDM layer of the Entity Framework. Because this it becomes part of the communication layer, behind the data container for interconnecting the various MS applications. Then how the ORM engine is internally implemented is not of so much importance. And I guess they will be more than happy at Microsoft if a third party ORM engine could be used, and build on top of the EDM and proving very efficient. It will just go in their direction of promoting the entity framework and especially the EDM layer and through the EDM the maturity of their platform and catalogue of products and they will be in a position to market and sell how easy interconnecting them is.

Event based architecture and test driven development

I listened today an interesting podcast related to event based systems and one aspect especially caught my attention: how an event based architecture is a good cadidate for test driven development.

You can find this podcast on DotNetRocks: "Ted Faison takes events to the limit" and the speaker is Ted Faison.

So first interesting point is the difference Ted makes between events based architectures and event driven architectures. Ted describes event driven applications as applications that react to events and therefore whose inputs are events but events are not necessarily used as a core design element of the application. With no events fired, event driven applications are just idle. On the other hand event based applications are applications that are architected and designed with events as a core technique to inter-connect application components. Events in event based architectures are really a first class citizen, design element of the system.

An interesting point is that in a properly designed event based architecture, the inner components of a system are loosely coupled, and you can even imagine changing connexion between components, either statically through configuration, or dynamically at runtime. And high level loss coupling in software architecture is definitely a high goal to aim at.

To design an event based applications, and enforce the components lose coupling and ability to map and inter-connect components in a configurable way, well approved patterns exists to do that, so an event based applications will most probably integrate patterns like builders, binders and coordinators. For instance, to achieve the highest level of loose coupling, neither the receiver of the event nor the emitter of the event should know of each other; and to achieve that, a binder component would have the responsibility to wire and connect the components.

So here we come to the very interesting part of Ted's talk: event based architectures are by essence well suited to test driven applications due to the lose coupling aspect they promote: you can imagine testing single components of your applications without having to bring up the whole application including databases, web context and all expensive or difficult to test resources. And to not bring the whole application up, the only thing you have to do is to simply not connect the components; or to connect to mock up components instead for testing purposes.

And it is commonly understood and approved now that TDD development is a very good practice to build robust, maintainable and flexible systems, even though quite often proving to be a big challenge. Go event based and you have a good starting point! I am personally using dependency injection patterns to build the loose coupled application architectures + dynamic mocking framework for test driven development (more on this on other posts). Event based architecture is another interesting strategy to achieve the same, while having its own advantages and draw-backs. So up to you to decide on one, or maybe any other alternative!

So as a conclusion, I must say that Ted Faison's podcast was very compelling and convincing and an enjoyable listening session, presenting a deep look on amazing possibilities open by using events! It could worth having a look at Ted's book now: Events based programming: Taking events to the limit.

REST Versus Web Services

Okay, I had been hearing a lot about REST lately, not really knowing what it was. Seemed like standard web stuff for me, lots of talks about HTTP verbs, post and get and that kind of things. So I wonder was what was new in the whole idea? I therefore decided to educate myself a bit and started with listening to this podcast of Software Engineering Radio: Stephen Tilkov on REST

So indeed there is a lot about just standard, existing web standards in REST, and this is what somehow makes it powerful. The new idea is to use standard web techniques to perform inter-communications between systems, in a SOA fashion. I won't enter into technical details and explanation on how REST works, first of all because I am not an expert at all and secondly because there is already plenty of documentation out there on the cloud. I would advise you listen to the podcast for a start if you are completely new on the subject. And good thing about this podcast is that the speaker is coming from the web services world and moved later to REST, he therefore has a good idea of the differences and pro and cons of both techniques.

 

My intent in this post is to try to analyze what REST means, where it apparently fits in the web and SOA world and what may o may not the proper scenarios to use it. I will do that in a very humble way, dropping my first thoughts on the matter. I may be missing important points and be inaccurate on some others so all comments are very much welcomed!

 

At first REST appears like a simplification of web services, SOAP and WSDL technologies to remotely connect systems in a system implementation technology independent fashion; there is obviously a technology common point in it and that's happening at the transport and protocol level, but internal technology used to implement the communicating systems can be anything as long as it is providing REST end points. I remember when in the past I was using DCOM to connect systems, thinking that such component technologies like DCOM or CORBA where the proper way to make distributed systems communicate and the de-facto standard (or multiple-standards I must say as they are not technology agnostics). I remembered then when I first heard about web services and SOAP and thought: hey what's that!! we have great, complex technologies like DCOM or similar (by the way I Loved COM...) to interconnect systems, so how can a simple kind of loose coupled messaging system could possibly do the trick instead and efficiently connect my systems?? Well today in state of the art architectures, inter-applications communication is (in most cases) handled through web services, at least for communication outside of the company, or between systems that need to be kept independents and probably implemented with different technologies or by different teams. So will REST become the new standard the same way web services did in the past years?

My understanding of REST is that it's mainly useful for CRUD like operations, even though you can add support for more custom verbs than the standard HTTP verbs (but then you lose part of the advantage of seamless and full integration on the web that you would have by using only standard web and http technology). The main purpose of REST is to see business entities as web resources and access them the same way you would access a web page or an image or any other resource through HTTP. Great thing about the idea is that there is nothing new except! If you know how to get an Http resource through its URL, you know how to get a customer entity through REST, isn't that great? It means that interconnecting applications could prove dramatically easy. However if you need a "real" business service that does more than just CRUD operations, then you better off, IMHO, keeping working with classical web services, WSDL and SOAP over HTTP (or other transport layer BTW). To summarize, my understanding is that for enterprise level, business applications, classical Web Services could still in many cases remain the best choice. Another scenario is the need for advanced enterprise wide services like security, confidentiality, transaction (even though transactions on the web has to be taking with care, we should rather be talking of compensation systems), asynchronous processing, then classical services will have much more to provide with the WS* implementations.

Another point to note with REST is that the communication is quite "relaxed". There is no notion of strong data type as there is with WSDL contracts. Therefore it's up to the client to know about the data that is received. If you need extra expressivity for your service then REST might not be the best choice. Well one can argue than REST can get business entities in an XML format, so you can add expressivity through schemas. Other issues I can foresee: how do you handle versioning of your services with REST? How do handle service location and service directories? Well there are many aspects to cover and would require more writing (and more investigation!) and will not be covered in this quick overview.

On the other side, for web facing services, REST will be a great solution: after all, most of the mashup web scenarios are using CRUD operations, and not even CRUD operation but simple read operations in many cases. REST is then much easier to use to mash up services than going through web services which need proxies to be accessed, and specific knowledge of SOAP and WSDL. And for what security is concerned, forget about all WS* specification: just use standard web security protocols, making interconnecting systems on the web in a secured way much easier and accessible to more people.

Well that was just my basic vision after first a first go through REST talks here and there, let's see in the future what comes up!

Oh, another wonder I had: hey! we're having this new, exciting feature of Microsoft.NET framework called WCF and it is all about connecting systems in a (as far as possible...) technology, protocol and transport independent way. So what about REST versus WCF?? Well good news is that WCF now allows to expose services through REST and consume REST services, so this is just a great thing, because one can invest on WCF without having to wonder if REST and a REST implementation would be a better choice: no worries: it's part of WCF through the new WebHttpBinding! So let's see how easy it is to build a WCF service, expose it through SOAP and with simple configuration change, expose it through REST. Haven't tried yet but should be pretty straightforward! We can see with this specific case an example of how powerful and extensible is WCF!

 

While we are talking about Microsoft and Microsoft technologies, I guess that REST will be interesting for Microsoft in many scenarios, let's see what they do and if they use REST for interconnecting their various systems. For instance we can already see coming the generalization of the use of the Entity Framework and especially the EDM (Entity Data Model) aspect for interconnecting Microsoft systems and exchanging data (more on another post). So REST would be a way to exchange the EDM entities across systems like Astoria Data services, Velocity, SharePoint and much more… And I was telling that REST seemed a good candidate for CRUD operations; well velocity and Astoria are all about exchanging data in a CRUD fashion, aren´t they? So let´s see how all will be consolidated in the future!

That´s it for this first overview of REST versus Web Services.

Object-Relational Mapping (ORM) - Overview

Today's talk is about object-relational mapping, what it is, why it is an essential part of software applications. I will then specifically talk (in a later post) about the ORM framework/tool I am regularly using on my projects and provide consultancy on to my clients: LlblgenPro

I don't pretend here to be an expert on ORM tools inner details and design complexity, I am posting here as an ORM tool user and express my return on experience on this matter.

     

It's all about data

The vast majority of software applications are dealing with data and data is the core business value of these applications. May you lose or corrupt this data and you are out of business. In some cases, data is an even more critical asset that also needs to be protected against theft and usage by malveillant people.

    

Data is relational

In the vast majority of cases data is stored in relational database systems. Reason for this is first historical: database systems have been developed over years, and evolved to very performing, scalable, reliable, widely used systems. For such a critical asset of any business, this is not something that will change easily, even though object databases have emerged and are used is some cases. Secondly, the relational representation of data along with the commonly used query language (SQL) are well suited to querying and consolidating large amount of data in a concurrent multi-user mode, computing aggregate information, insuring transactional operations and enforcing integrity of data.

     

Code is object oriented

Object oriented development is now widely accepted and used for its well-known benefits that we will not detail here. Through object oriented design and techniques, code is made a close abstraction of real world and business domain concepts. Many design patterns have been identified that help solve common design problems and leverage the power of object oriented design to build robust, flexible solutions.

    

Impedance mismatch and data access layer

Now we can spot the problem/challenge and the mismatch that exists between application code that is object oriented and need to work on data represented and stored in a relational format. And from a separation of concern point of view we want this mismatch to be encapsulated and abstracted from both the database layer and the application code layer. It is indeed a best practice from a software architecture point of view to separate concerns and loose couple the various aspects of a software project (more on this in a future post). In our specific case we want to have an independent relational database layer created, maintained and optimized by a database specialist who does not need to be concerned at all by the way his data will be accessed and used; and at the same time ensure that application developers can use their best creativity and knowledge on how to efficiently build flexible, robust and maintainable application code in an object oriented fashion without having to deal with the technical intricacy of converting back and forth relational data to objects. And this is where we introduce the data access layer, which purpose is to encapsulate all the tricky details and logic needed to perform the bi-directional transformation between relational data and object oriented application data. From now on, we will name object oriented application data after the denomination of business objects or entities.

          

OK, so can't I develop my own data access layer?     

As developers, most of us have one day developed our own data access layer, taking care of communicating with the database, retrieving and exposing data to the business layers and performing the common CRUD (create/update/delete) actions against the database. After all, this at first does not seem like such a big deal does it? But it very quickly appears that writing proper, efficient and robust data access code is really a hell of a work. For instance how about managing primary keys / foreign keys relationship? How about dealing with strongly typed data rather than simple raw data tables? What about server side pagination? What about complex retrieval of data across multiple tables?

Other fact is that the data access code is highly technical plumbing code that is good candidate to be made generic and thus legitimate the use of a common set of tool, framework and runtime libraries to perform the work for us and concentrate on the core business implementation. So here we are: after writing one's own data access layer on a real, reasonably sized project, one will quite fast conclude that leveraging an existing ORM tool is relevant!

     

So tell me about ORM tools!

One important point to note is that a data access layer does not necessarily have to be ORM layer; in other words, a basic data access layer that only provide encapsulation of the database connections, database commands building and requesting data that is then used by the business layer (probably as ADO.NET datasets) is not a proper ORM layer. The ORM layer goes one step further than a basic data access layer by providing (or connecting to, through a mapping configuration) a business entity model layer on top of the actual data access code and most probably advanced functionalities that can include various interesting stuff like object oriented query language (that could now be exposed through the Microsoft Link extension to .NET languages), web service and distributed application support, auditing, database schema change management, multi-databases, support for data-binding etc.

Using an ORM can therefore save an awful lot of work and give you for free many extra functionality that you can potentially use to improve or extend your applications. The only matter then is to select one ORM tool from the many existing.     

You can see from what exists that ORM tools can be of different nature, you will find:

1. Simple entity objects and data access code generator (which I believe are not really fully fledged and powerful ORM tools). Example could be Code Smith templates.

2. Entity object layer code generator + runtime libraries for data access. Example of such a tool is LlblgenPro.

3. Mapping configuration to connect an existing data store to an existing entity object layer + runtime libraries for data access. Example of such a tool is NHibernate

 

Please note that an ORM tool can also be data centric, model centric or both. A data centric ORM tool will need an existing database schema and generate its entity business model from it; a model centric ORM tool will be able to generate a database from an object entity model. And hybrids ORM tool will be able to connect an existing entity object model and existing database through a configuration mapping file. 

I don't personally find the model centric approach and generation of database relevant. The database should IMHO be designed separately. There are techniques to design efficient well performing databases and I don't believe an automatic generation of the schema is appropriate, except maybe for quick proof of concept prototype or very low load applications. Designing an appropriate database schema is too much dependant on the application business domain and the usage that will be done of it, and the load that will be supported by the application.  

I personally find the data centric approach being the best compromise for most applications. The constraint is that your business model should then be quite close to your data model, which I believe in most business application is the case. It's maybe not the case for warehouse databases designed and optimized for reporting purposes, but there are other techniques than business application development to manage their data and more a appropriate scenario would then to use a specialized data reporting server application (like Microsoft SQL Reporting Services). Another case for which the entity data model might not fit the database schema could be very high load databases for which the database schema has been deeply optimized for performance.

I can see too ways to overcome the constraint of having close entity model and database schema and allowing some divergence between the models. One is to use a hybrid ORM tools allowing connection though mapping configuration between an entity object model and a database schema. However what I don't like about this solution is that you have to write your own entity data model where a data centric ORM tool could automatically generate it for you. Other option is to use a data centric ORM tool and add to the database schema database views that could encapsulate and hide for you the differences between your models. Well of course then you have to handle updateable view or use triggers to manage write operations on the view but hey, that would be the subject of another post!

  

So you now know my preferences. Of course every project has its own specificities and constraints and the choice of an ORM strategy must be revised for every new project. But I always try to save myself the most development work I could, and therefore anytime I can I go for a data centric ORM, because of its ability to generate for you both the entity object model and the actual data access layer, saving you a huge development effort! And my favorite ORM tool, of which I will talk about in a future post, is LlblgenPro.

About my Blog

The intent of this post is to describe the main contents and goals of my blog and what I will be posting about.

   

Main Topic and scope

As the title of my Blog states, the main topic covered is "Business Oriented Software Solutions". It's all about delivering fast time to market, good quality software that answers to business needs and result in a return on investment. 

The type of software projects that are in scope of what I call "Business Oriented Software Solutions" are commercial projects, and are not included research & development, educational projects and any other projects for which there is no associated direct business profitable value. However, all posts that are technical and about development methodologies and quality are still applicable to all projects.

Also not included in "Business Oriented Software Solutions" are all RAD projects and disposable software applications. Disposable software applications can be proof of concept prototypes or software application that are intended for a single, on-off purpose, and discarded away after use. "Business Oriented Software Solutions" are software applications that are designed and intended to be used on the long term and will most probably have to evolve and integrate new or changed business needs.   

This Blog being about software development, there will be a great deal of technology talks. Even though part of the posts will be technology agnostic, another part will be specific to Microsoft and Microsoft partners technologies. This is because I made the choice of using Microsoft centric technologies for the implementation of my software projects and it is therefore the technology platform I am expert on and will be posting about.

   

Target audience

This blog will cover all activities of a software development project and therefore is targeted to multiple audiences. Depending on your level of implication in software projects, you might be interested in certain posts only. Click on your profile below to filter posts of most interest to you.

   

Posts for business owners: As a business owner you will be interested in posts that reunite software solutions and business and explain how software solutions can bring your business added value and exactly fit to your business rather than your business be constrained and limited by a software solution. These posts do not require specific technical knowledge and yet give a nice and clear understanding on software and technical matters from a business perspective.

   

Posts for technical project managers: As a technical project manager you will be interested in all technical and process & development practices posts

   

Posts for functional project managers: As a functional project manager you will be interested in process & development methodology posts and business related posts. You will be especially interested to read about how to drive a project in a feature driven way and through highly iterative delivery cycles.

   

Posts for developers: As a developer you will be interested in all technical posts and in the process & development methodology posts that are related to your daily activities. May be of interest for you two to quickly go through all other posts to get an overall view on how you job fit in the total project landscape.

   

Posts for software architects: As a software architect you will be interested in all technical posts and especially these posts related to software architecture, development practices, tools and frameworks and design patterns.

    

Overview  

This blog is about "Business Oriented Software Solutions" or: how to deliver fast time to market, good quality software that answers to business needs and therefore result in a return on investment.  

This could seem quite obvious and not relevant to be the object of a complete blog: after all, who wants to build and late deliver bad quality software that is useless, not driven by any business case and not bringing any added value? Well, yes, one could tell that software development can be an exclusively creative and fulfilling, passionate activity for some individuals, but as stated above, less focus here on software developed as part of a commercial company strategy. And still: as a passionate individual you may don't care of your creation bringing any added business value and keep it useless yet exciting, but you most probably still want to efficiently produce high quality code!

So obvious answer is: NO, of course no-one wants to create these useless, expensive, poor quality software applications, so why this blog?? Thing is that developing and delivering these Business Oriented Solutions can prove to be tricky and the purpose of my Blog is to gather and consolidate information to provide guidance on how to best choose, use, re-use and compose processes, technologies, patterns, frameworks, tools and applications to deliver fast time to market, feature oriented, high quality, robust and performing software solutions.

I believe that the key to efficiently build Business Oriented Software Solution is to concentrate and excel in two complementary directions:

• Define the goals to aim at and set these goals as your priority
• Clearly understand the various areas of expertise you need to cover to achieve your goals and make sure you have skilled individual on board in each of these areas. 

And for our matter, Building Business Oriented Software Solutions is a real big challenge and the difficulty is to succeed in three main goals: create solutions that are business driven, high quality and fast time to market. 

And building Business Oriented Software Solutions involve mastering complementary skills in three areas of expertise: the business domain/functional space, technology & development practices and management & project methodologies.

  

The three goals for success

One can list more goals to aim at to successfully deliver Business Oriented Software Solutions, and this list can depend on the specificity of a project and/or business domain. However I believe than the main three goals to achieve that are essential and drive success in most cases are to build solutions that are:  

• Business driven/feature driven: solutions answering a real business need and seamlessly adapting to the business changes and evolutions.
• High quality: solutions technically properly designed and implemented making the solution robust, performing, maintainable and seamlessly welcoming evolutions and new features.
• Fast time to market: solution that have the ability to be delivered or more be iteratively delivered as fast as possible, in order to compete efficiently on the market place.

     

The difficulty comes from the interdependency between these three targets. It is for instance easy to understand that reaching fast time to market result while keeping a high level of quality is not so easy to achieve. This is part of the reason why it's common place to have software that are early on the market place but of poor quality, or software that are of good quality, delivered on time but not aligned with the business requirements and priorities, or even worth: software that is not answering the business initial needs, of poor quality due to the pressure involved by fast time to market constraint and delivered very late as a consequence of the two first problems.

The posts will be written with this in mind and will be oriented toward solutions and guidance to succeed in meeting the three targets.

     

The three areas of expertise involved

Various disciplines are to be conducted as part of as software project and these disciplines require different set of skills and abilities that we can distribute in three areas of expertise:  

• Business domain & Functional space: Know and understand the functional space and business domain that is yours. Should not be constrained by technology or methodologies, as business need is the driving force.
• Technology & development practices: Excel in efficiently using technology and composing the right frameworks, tools and patterns to answer a business need in a robust and flexible way.
• Management and project methodologies: Master appropriate management and project methodologies to efficiently and iteratively delivery software with accordance to business needs and technology constraints.

   

Please note that in a software project there are more disciplines than areas of expertise. If we take RUP as an example, the disciplines are: business modeling, requirements, analysis & design, implementation, test, deployment, configuration & change management, project management and environment. RUP is the IBM iterative software development process (http://en.wikipedia.org/wiki/IBM_Rational_Unified_Process) .

You need to be aware that no matter how small or big is your team you will have to master skills in all these three areas of expertise. And the challenge in succeeding in all three areas lies in the inherent difficulty of cross-cutting these three distinct aspects of a software development project that require complete different abilities and most usually different persons who need to communicate and understand each-other and each-other concerns.

      

Outcome of the posts  

Each of the posts that are relevant to creating Business Oriented Software Solutions will be started with tables indicating what goals the discussed topic can help achieve and what area of expertise is concerned and the posts will be tagged accordingly. This way the reader will have a mapping between goals and areas of expertise and a set of relevant posts, ideas, and guidance to succeed in efficiently building Business Oriented Software Solutions.

So here we start, I will now start blogging, I hope you will enjoy the ride!