Walter Almeida's Blog

Is the Microsoft Entity Framework more than just an ORM?

I recently listened to the show #369 of .Net Rocks! And read the reaction of Frans Bourma (LLBLGenPro) on the subject.

I wanted to react and give my views on the subject. 

For an overview of what Object Relational Mapping is, please read my previous post on the subject.  

This post assumes that the reader will already have the knowledge of the Microsoft Entity Framework concepts. If it is not the case please read the ADO.NET Entity Framework Overview as a pre-requisite.

Quick introduction to the Microsoft Entity Data Model (EDM)   

The MS Entity Framework is composed of several parts, one of which is the Entity Data Model (EDM). The EDM is used for modelling at a conceptual level the business space, and actually corresponds to the "object space" of the Object/Relational mapping. The EDM is the model describing the set of objects mapped to relational tables on the database side. Then comes the mapping layer, being the bridge or glue between the EDM, or "object side" and the tables in your relational database. Finally, the Object Relational Mapping engine does the trick to manipulate data, persist changes to the database and feed the EDM objects at runtime. An important aspect that will be essential for the main subject of the post is the following: the EDM model is decoupled from the mapping layer and from the ORM engine. In other words the EDM model has no knowledge that there is a mapping and an ORM engine at all, and can be used or re-used without the presence of the mapping and ORM engine. This is an essential pattern for working in a remote/disconnected scenario where a service is responsible for manipulating and persisting data (and thus is using the mapping layer and ORM engine), and the client solely use the data provided by the services, through the form of EDM objects, with no knowledge on how and where the data is stored. Please note that there is nothing conceptually new here and this is the basis of most existing ORM tools of the market, adapted to the Microsoft terminology.

So now that the introduction is made, here comes the main subject of this post.

The Microsoft Entity Framework entering the ORM tools market

For what I've understood from the .Net Rocks podcast, Microsoft could be having more interest on the EDM model than on the ORM engine. You could imagine that this is because they did not manage to catch-up with the existing ORM tools maturity or for a more strategic reason, but that's out of subject here.

Fact is they are many ORM vendors, doing and mastering this job for years and being very mature. The EDM specifically could be on its side more interesting for MS because it is platform oriented and could become a new Microsoft standard, ideal for making applications inter-communicate though this EDM full object oriented, strongly typed object model. Let be clear: the Entity Framework is not a platform, however the Entity Framework and more specifically the EDM model can become an important glue element of the Microsoft platform, becoming the de-facto standard for inter-connecting Microsoft applications and services and any third party application wanting to leverage the EDM opportunity and offer a communication interface to the outer world though EDM entities.

  

A new business opportunity for third party vendors?

What I would extrapolate from Stephen Forte talk in the podcast (and that engages only me) is that there is an opportunity for ORM vendors to build on top of EDM and provide advanced and efficient ORM engine core functionalities: a third party ORM engine replacing the Microsoft one, manipulating and persisting data all through EDM entities. There is also an opportunity for Microsoft Partners to build tools or designers that could assist creating a EDM model and map it to the database + making use of an ORM engine (being the Microsoft one or any other "EDM Compliant" ORM engine). In this scenario, third parties ORM vendors would be using the EDM as the data model used to manipulate, transfer data, between applications and services. Another clarification: the EDM is indeed not a new, revolutionary concept, and every ORM tool in the market is including such a concept: the entities model is indeed the actual object oriented side of the object/relation mapping layer and is an essential part of any ORM tool.

    

Closer look on the EDM on a broader picture

The EDM model becomes a data contract in the WCF sense, making easier and efficient to exchange data between systems, regardless of where the underlying data is stored and whether or not we are using ORM engine to concretely retrieve these EDM entities. For instance you could most probably use and EDM model to represent data in an inter-application communication scenario and not link to any database in the back-end. The EDM and the Entity Framework ORM engine are loose-coupled and you can use one or the other independently. Well at least you can use the EDM independently from the EF ORM engine, I am not too sure of the opposite.

And the EDM is even more than just a data contract: it comes with tools and will probably come with more. One is the Entity SQL, based on SQL syntax oriented toward object queries on the EDM model. And again this is nothing new: for instance Llblgen provides a full framework around its entity model, adding extra functionalities as compared to a basic object model or to any model you could build from scratch by hand, in a reasonable amount of time.

So back to the point: using the EDM as a data contract, it removes the need for applying a DTO pattern for interoperability between systems and especially when exposing services and wanting to make your clients unaware and decoupled from your inner implementation and especially from the ORM tool you are using. Well to be more precise: if the EDM becomes a "Microsoft standard", any one agreeing on that could consume a service delivering EDM entities. But you should be aware that the interoperability will anyway not be complete as using EDM will probably tight you to the Microsoft platform and "Microsoft Entity Framework aware" clients. I will write another post about the DTO pattern in a SOA world and when to use or not use it. DTO is yet another mapping layer and one can wonder why after mapping from the relational world to the object world we still need another translation. My recommendation would be that one should not introduce another mapping layer is there is no absolute need for it. You might need it or might absolutely want to forget about it, depending on the scenario you're in, there is no single solution to all problems.

So is the Entity Data Model much bigger than a just an ORM?

So Yes, for what the EDM is concerned and what you can hear in the podcast as being "more than just ORM", you can achieve the same result with other ORM tool and I am actually doing it on my projects with the LLBLGen Pro generated entities model layer. What however the EDM might provide as a plus in the future is integration in the whole MS suite and become the missing glue link to seamlessly connect systems though the strongly type, object oriented EDM data model. Well let´s be more precise: the missing link in systems based on Microsoft tools and technology.   

So to insist on this point, I can understand Frans Bourma when he is reacting strongly on the sentence "The EF is much more than an ORM" because no, there is no magic and what MS say they can do with the entity framework and the EDM could be achieved with another ORM tool and its own entity model. It is just after all a matter of convention and all actors standardizing on the same framework (and for instance here the EDM part of entity framework). But Yes MS can bring something more with the EDM that goes beyond the ORM aspect, just by the fact that they can push for using the EDM as the data contract layer for all of their products. They can do it because they own the product! And this is what makes the idea powerful: for instance when a company start investing in Microsoft, it usually spread the whole SI and range of applications: start installing MS servers and desktops windows OS and you end up using Exchange server and SharePoint server and office and more and more, rather that competing products. And why? Just because all integrate so well: it is all coming for a single vendor who actually makes the integration of all its range of applications as easy and straight-forward as possible! Same thing can happen with the EDM on a more technical, low level I must say, but following the same driving idea.

      

To conclude …   

To conclude, what I can foresee is that what will bring most to Microsoft strategically speaking is the EDM layer of the Entity Framework. Because this it becomes part of the communication layer, behind the data container for interconnecting the various MS applications. Then how the ORM engine is internally implemented is not of so much importance. And I guess they will be more than happy at Microsoft if a third party ORM engine could be used, and build on top of the EDM and proving very efficient. It will just go in their direction of promoting the entity framework and especially the EDM layer and through the EDM the maturity of their platform and catalogue of products and they will be in a position to market and sell how easy interconnecting them is.

Event based architecture and test driven development

I listened today an interesting podcast related to event based systems and one aspect especially caught my attention: how an event based architecture is a good cadidate for test driven development.

You can find this podcast on DotNetRocks: "Ted Faison takes events to the limit" and the speaker is Ted Faison.

So first interesting point is the difference Ted makes between events based architectures and event driven architectures. Ted describes event driven applications as applications that react to events and therefore whose inputs are events but events are not necessarily used as a core design element of the application. With no events fired, event driven applications are just idle. On the other hand event based applications are applications that are architected and designed with events as a core technique to inter-connect application components. Events in event based architectures are really a first class citizen, design element of the system.

An interesting point is that in a properly designed event based architecture, the inner components of a system are loosely coupled, and you can even imagine changing connexion between components, either statically through configuration, or dynamically at runtime. And high level loss coupling in software architecture is definitely a high goal to aim at.

To design an event based applications, and enforce the components lose coupling and ability to map and inter-connect components in a configurable way, well approved patterns exists to do that, so an event based applications will most probably integrate patterns like builders, binders and coordinators. For instance, to achieve the highest level of loose coupling, neither the receiver of the event nor the emitter of the event should know of each other; and to achieve that, a binder component would have the responsibility to wire and connect the components.

So here we come to the very interesting part of Ted's talk: event based architectures are by essence well suited to test driven applications due to the lose coupling aspect they promote: you can imagine testing single components of your applications without having to bring up the whole application including databases, web context and all expensive or difficult to test resources. And to not bring the whole application up, the only thing you have to do is to simply not connect the components; or to connect to mock up components instead for testing purposes.

And it is commonly understood and approved now that TDD development is a very good practice to build robust, maintainable and flexible systems, even though quite often proving to be a big challenge. Go event based and you have a good starting point! I am personally using dependency injection patterns to build the loose coupled application architectures + dynamic mocking framework for test driven development (more on this on other posts). Event based architecture is another interesting strategy to achieve the same, while having its own advantages and draw-backs. So up to you to decide on one, or maybe any other alternative!

So as a conclusion, I must say that Ted Faison's podcast was very compelling and convincing and an enjoyable listening session, presenting a deep look on amazing possibilities open by using events! It could worth having a look at Ted's book now: Events based programming: Taking events to the limit.

REST Versus Web Services

Okay, I had been hearing a lot about REST lately, not really knowing what it was. Seemed like standard web stuff for me, lots of talks about HTTP verbs, post and get and that kind of things. So I wonder was what was new in the whole idea? I therefore decided to educate myself a bit and started with listening to this podcast of Software Engineering Radio: Stephen Tilkov on REST

So indeed there is a lot about just standard, existing web standards in REST, and this is what somehow makes it powerful. The new idea is to use standard web techniques to perform inter-communications between systems, in a SOA fashion. I won't enter into technical details and explanation on how REST works, first of all because I am not an expert at all and secondly because there is already plenty of documentation out there on the cloud. I would advise you listen to the podcast for a start if you are completely new on the subject. And good thing about this podcast is that the speaker is coming from the web services world and moved later to REST, he therefore has a good idea of the differences and pro and cons of both techniques.

 

My intent in this post is to try to analyze what REST means, where it apparently fits in the web and SOA world and what may o may not the proper scenarios to use it. I will do that in a very humble way, dropping my first thoughts on the matter. I may be missing important points and be inaccurate on some others so all comments are very much welcomed!

 

At first REST appears like a simplification of web services, SOAP and WSDL technologies to remotely connect systems in a system implementation technology independent fashion; there is obviously a technology common point in it and that's happening at the transport and protocol level, but internal technology used to implement the communicating systems can be anything as long as it is providing REST end points. I remember when in the past I was using DCOM to connect systems, thinking that such component technologies like DCOM or CORBA where the proper way to make distributed systems communicate and the de-facto standard (or multiple-standards I must say as they are not technology agnostics). I remembered then when I first heard about web services and SOAP and thought: hey what's that!! we have great, complex technologies like DCOM or similar (by the way I Loved COM...) to interconnect systems, so how can a simple kind of loose coupled messaging system could possibly do the trick instead and efficiently connect my systems?? Well today in state of the art architectures, inter-applications communication is (in most cases) handled through web services, at least for communication outside of the company, or between systems that need to be kept independents and probably implemented with different technologies or by different teams. So will REST become the new standard the same way web services did in the past years?

My understanding of REST is that it's mainly useful for CRUD like operations, even though you can add support for more custom verbs than the standard HTTP verbs (but then you lose part of the advantage of seamless and full integration on the web that you would have by using only standard web and http technology). The main purpose of REST is to see business entities as web resources and access them the same way you would access a web page or an image or any other resource through HTTP. Great thing about the idea is that there is nothing new except! If you know how to get an Http resource through its URL, you know how to get a customer entity through REST, isn't that great? It means that interconnecting applications could prove dramatically easy. However if you need a "real" business service that does more than just CRUD operations, then you better off, IMHO, keeping working with classical web services, WSDL and SOAP over HTTP (or other transport layer BTW). To summarize, my understanding is that for enterprise level, business applications, classical Web Services could still in many cases remain the best choice. Another scenario is the need for advanced enterprise wide services like security, confidentiality, transaction (even though transactions on the web has to be taking with care, we should rather be talking of compensation systems), asynchronous processing, then classical services will have much more to provide with the WS* implementations.

Another point to note with REST is that the communication is quite "relaxed". There is no notion of strong data type as there is with WSDL contracts. Therefore it's up to the client to know about the data that is received. If you need extra expressivity for your service then REST might not be the best choice. Well one can argue than REST can get business entities in an XML format, so you can add expressivity through schemas. Other issues I can foresee: how do you handle versioning of your services with REST? How do handle service location and service directories? Well there are many aspects to cover and would require more writing (and more investigation!) and will not be covered in this quick overview.

On the other side, for web facing services, REST will be a great solution: after all, most of the mashup web scenarios are using CRUD operations, and not even CRUD operation but simple read operations in many cases. REST is then much easier to use to mash up services than going through web services which need proxies to be accessed, and specific knowledge of SOAP and WSDL. And for what security is concerned, forget about all WS* specification: just use standard web security protocols, making interconnecting systems on the web in a secured way much easier and accessible to more people.

Well that was just my basic vision after first a first go through REST talks here and there, let's see in the future what comes up!

Oh, another wonder I had: hey! we're having this new, exciting feature of Microsoft.NET framework called WCF and it is all about connecting systems in a (as far as possible...) technology, protocol and transport independent way. So what about REST versus WCF?? Well good news is that WCF now allows to expose services through REST and consume REST services, so this is just a great thing, because one can invest on WCF without having to wonder if REST and a REST implementation would be a better choice: no worries: it's part of WCF through the new WebHttpBinding! So let's see how easy it is to build a WCF service, expose it through SOAP and with simple configuration change, expose it through REST. Haven't tried yet but should be pretty straightforward! We can see with this specific case an example of how powerful and extensible is WCF!

 

While we are talking about Microsoft and Microsoft technologies, I guess that REST will be interesting for Microsoft in many scenarios, let's see what they do and if they use REST for interconnecting their various systems. For instance we can already see coming the generalization of the use of the Entity Framework and especially the EDM (Entity Data Model) aspect for interconnecting Microsoft systems and exchanging data (more on another post). So REST would be a way to exchange the EDM entities across systems like Astoria Data services, Velocity, SharePoint and much more… And I was telling that REST seemed a good candidate for CRUD operations; well velocity and Astoria are all about exchanging data in a CRUD fashion, aren´t they? So let´s see how all will be consolidated in the future!

That´s it for this first overview of REST versus Web Services.

Object-Relational Mapping (ORM) - Overview

Today's talk is about object-relational mapping, what it is, why it is an essential part of software applications. I will then specifically talk (in a later post) about the ORM framework/tool I am regularly using on my projects and provide consultancy on to my clients: LlblgenPro

I don't pretend here to be an expert on ORM tools inner details and design complexity, I am posting here as an ORM tool user and express my return on experience on this matter.

     

It's all about data

The vast majority of software applications are dealing with data and data is the core business value of these applications. May you lose or corrupt this data and you are out of business. In some cases, data is an even more critical asset that also needs to be protected against theft and usage by malveillant people.

    

Data is relational

In the vast majority of cases data is stored in relational database systems. Reason for this is first historical: database systems have been developed over years, and evolved to very performing, scalable, reliable, widely used systems. For such a critical asset of any business, this is not something that will change easily, even though object databases have emerged and are used is some cases. Secondly, the relational representation of data along with the commonly used query language (SQL) are well suited to querying and consolidating large amount of data in a concurrent multi-user mode, computing aggregate information, insuring transactional operations and enforcing integrity of data.

     

Code is object oriented

Object oriented development is now widely accepted and used for its well-known benefits that we will not detail here. Through object oriented design and techniques, code is made a close abstraction of real world and business domain concepts. Many design patterns have been identified that help solve common design problems and leverage the power of object oriented design to build robust, flexible solutions.

    

Impedance mismatch and data access layer

Now we can spot the problem/challenge and the mismatch that exists between application code that is object oriented and need to work on data represented and stored in a relational format. And from a separation of concern point of view we want this mismatch to be encapsulated and abstracted from both the database layer and the application code layer. It is indeed a best practice from a software architecture point of view to separate concerns and loose couple the various aspects of a software project (more on this in a future post). In our specific case we want to have an independent relational database layer created, maintained and optimized by a database specialist who does not need to be concerned at all by the way his data will be accessed and used; and at the same time ensure that application developers can use their best creativity and knowledge on how to efficiently build flexible, robust and maintainable application code in an object oriented fashion without having to deal with the technical intricacy of converting back and forth relational data to objects. And this is where we introduce the data access layer, which purpose is to encapsulate all the tricky details and logic needed to perform the bi-directional transformation between relational data and object oriented application data. From now on, we will name object oriented application data after the denomination of business objects or entities.

          

OK, so can't I develop my own data access layer?     

As developers, most of us have one day developed our own data access layer, taking care of communicating with the database, retrieving and exposing data to the business layers and performing the common CRUD (create/update/delete) actions against the database. After all, this at first does not seem like such a big deal does it? But it very quickly appears that writing proper, efficient and robust data access code is really a hell of a work. For instance how about managing primary keys / foreign keys relationship? How about dealing with strongly typed data rather than simple raw data tables? What about server side pagination? What about complex retrieval of data across multiple tables?

Other fact is that the data access code is highly technical plumbing code that is good candidate to be made generic and thus legitimate the use of a common set of tool, framework and runtime libraries to perform the work for us and concentrate on the core business implementation. So here we are: after writing one's own data access layer on a real, reasonably sized project, one will quite fast conclude that leveraging an existing ORM tool is relevant!

     

So tell me about ORM tools!

One important point to note is that a data access layer does not necessarily have to be ORM layer; in other words, a basic data access layer that only provide encapsulation of the database connections, database commands building and requesting data that is then used by the business layer (probably as ADO.NET datasets) is not a proper ORM layer. The ORM layer goes one step further than a basic data access layer by providing (or connecting to, through a mapping configuration) a business entity model layer on top of the actual data access code and most probably advanced functionalities that can include various interesting stuff like object oriented query language (that could now be exposed through the Microsoft Link extension to .NET languages), web service and distributed application support, auditing, database schema change management, multi-databases, support for data-binding etc.

Using an ORM can therefore save an awful lot of work and give you for free many extra functionality that you can potentially use to improve or extend your applications. The only matter then is to select one ORM tool from the many existing.     

You can see from what exists that ORM tools can be of different nature, you will find:

1. Simple entity objects and data access code generator (which I believe are not really fully fledged and powerful ORM tools). Example could be Code Smith templates.

2. Entity object layer code generator + runtime libraries for data access. Example of such a tool is LlblgenPro.

3. Mapping configuration to connect an existing data store to an existing entity object layer + runtime libraries for data access. Example of such a tool is NHibernate

 

Please note that an ORM tool can also be data centric, model centric or both. A data centric ORM tool will need an existing database schema and generate its entity business model from it; a model centric ORM tool will be able to generate a database from an object entity model. And hybrids ORM tool will be able to connect an existing entity object model and existing database through a configuration mapping file. 

I don't personally find the model centric approach and generation of database relevant. The database should IMHO be designed separately. There are techniques to design efficient well performing databases and I don't believe an automatic generation of the schema is appropriate, except maybe for quick proof of concept prototype or very low load applications. Designing an appropriate database schema is too much dependant on the application business domain and the usage that will be done of it, and the load that will be supported by the application.  

I personally find the data centric approach being the best compromise for most applications. The constraint is that your business model should then be quite close to your data model, which I believe in most business application is the case. It's maybe not the case for warehouse databases designed and optimized for reporting purposes, but there are other techniques than business application development to manage their data and more a appropriate scenario would then to use a specialized data reporting server application (like Microsoft SQL Reporting Services). Another case for which the entity data model might not fit the database schema could be very high load databases for which the database schema has been deeply optimized for performance.

I can see too ways to overcome the constraint of having close entity model and database schema and allowing some divergence between the models. One is to use a hybrid ORM tools allowing connection though mapping configuration between an entity object model and a database schema. However what I don't like about this solution is that you have to write your own entity data model where a data centric ORM tool could automatically generate it for you. Other option is to use a data centric ORM tool and add to the database schema database views that could encapsulate and hide for you the differences between your models. Well of course then you have to handle updateable view or use triggers to manage write operations on the view but hey, that would be the subject of another post!

  

So you now know my preferences. Of course every project has its own specificities and constraints and the choice of an ORM strategy must be revised for every new project. But I always try to save myself the most development work I could, and therefore anytime I can I go for a data centric ORM, because of its ability to generate for you both the entity object model and the actual data access layer, saving you a huge development effort! And my favorite ORM tool, of which I will talk about in a future post, is LlblgenPro.