Walter Almeida's Blog

Introducing my favorite ORM Tool: LLBLGEN - Part 2

Ok, time to write again about LLBLGen, my favorite ORM tool. It has been my favorite ORM for years now and I did write a post on the subject several years ago: http://blog.walteralmeida.com/2008/09/introducing-my-favorite-orm-tool-llblgen.html - So this is why I call this post "Introducing my favorite ORM Tool: LLBLGEN - Part 2" :)

For information, I will talk only about the new features that were of higher interest to me. There's much more, so check out Llblgen web site for more info: http://www.llblgen.com

So here we go: Llblgen is now more than never still my favorite tool: since version v3.0; the tools supports several well-known ORM frameworks: Entity Framework, Linq to SQL, NHibernate and of course the Llblgen framework. You now have to a single designer that let you build a data access layer using the ORM framework of your choice. For information, I am still using the Llblgen framework, can’t help it…

And IMHO, the most amazing new feature of all is… the support of model-first development! This was a must have. In the previous version, you had to start from an existing database schema to build your entity model and data access layer on top of it. It is no more the case: you gain in abstraction and can actually start from your entity model! The tool will generate for you the database schema on the target database of your choice, all based on your model. The tool is even capable of generating for you update scripts to integrate your entity model changes to the database: modification, creation and deletion scripts.

Important also to realize that you don’t lose the database first feature: shall you want to build on top of an existing database, or keep full control of your database schema: you still can start from your existing database. And… you can actually mix both and build a part of the data layer in a model first approach and build the rest following a database first approach, which could be a great option if you want to extend an existing database or keep control of mission critical sections of the database schema. You can also start database first and switch later to model first or the other way round. Let’s say: it is very flexible and implemented in a very loose coupled way making all these scenarios possible. I guess this will keep everybody happy and give the option to big believers of one or the other of the approaches to give a try to the other one!

Ok just one word about my view: I am a big believer of the model first approach, in a general way, that is: not only for the data layer, but for full enterprise application. I was therefore very happy when Llblgen V3 came to life!

What else could I say… Oh yes: you now have a graphical designer (called model views) allowing you to build entity models in a very user friendly way. It is also a must have to share with business people on the design of your business model.

What’s more? Full XML project file, which proves very useful to properly integrate Llblgen into your development lifecycle. Indeed, if you follow a model first approach, you then have to consider the model (here the Llblgen project file) as a first class citizen artifact of your project and therefore you should be able to compare versions of your models between several iteration. This is mandatory for proper design reviews. Well no problem to use your standard tooling on top of the Llblgen XML file!

In terms of code and product quality, no need to say that it is top quality, anyone who knows Frans Bouma will understand what I am saying:) No comment: great code, both the generated code and the code base. Nice and neat and effective and bug-free. Well there is no bug-free code, but this code could be considered bug-free when comparing to many other small and big companies. Even the V3 Beta 1 version was already of high quality. And this is essential for a tool that generates code and is targeted at developers. And if you manage to point out a bug: you'll get great first support and have a fix pretty soon available! To be honest: I was much more happy with Llblgen Support than with IBM support... 

Ok now my favorite: extensibility. Llblgen Pro is highly extensible in many ways. And extensions can be written by building on top of the Llblgen SDK. BTW this is actually where you get an even more clear idea of the quality of the underlying Llblgen code. And this is essential for today’s software applications to have this extensibility capability because this is how an eco-system can be built around. In that context, as part of FENOMEN Project (www.fenomen.pro), we have been building for the last two years innovative products that allow the modeling and generation of full enterprise, business applications. And we built on top of Llblgen Pro which is our core ORM framework and entity model designer. This use of Llblgen was essential to quick-start our research and development efforts and build products that really add value to existing market solutions. Indeed why creating what’s already there and took years and years of work to others to master? We are big believers of the “compose and reuse” paradigm: never create what’s already out there: reuse and compose the best of what’s on the market today, and build only what does not exist yet! So stay tuned, you’ll hear about FENOMEN the year to come and will see a great example of the potential of Lllblgen extensibility. And for all whom already now, use and love LLBLGEN, you’ll get a great opportunity to go much further than your best expectations when using LLBLGEN with FENOMEN extensions. Curious? More information in 2011!

 

 

 

 

 

 

Is the Microsoft Entity Framework more than just an ORM?

I recently listened to the show #369 of .Net Rocks! And read the reaction of Frans Bourma (LLBLGenPro) on the subject.

I wanted to react and give my views on the subject. 

For an overview of what Object Relational Mapping is, please read my previous post on the subject.  

This post assumes that the reader will already have the knowledge of the Microsoft Entity Framework concepts. If it is not the case please read the ADO.NET Entity Framework Overview as a pre-requisite.

Quick introduction to the Microsoft Entity Data Model (EDM)   

The MS Entity Framework is composed of several parts, one of which is the Entity Data Model (EDM). The EDM is used for modelling at a conceptual level the business space, and actually corresponds to the "object space" of the Object/Relational mapping. The EDM is the model describing the set of objects mapped to relational tables on the database side. Then comes the mapping layer, being the bridge or glue between the EDM, or "object side" and the tables in your relational database. Finally, the Object Relational Mapping engine does the trick to manipulate data, persist changes to the database and feed the EDM objects at runtime. An important aspect that will be essential for the main subject of the post is the following: the EDM model is decoupled from the mapping layer and from the ORM engine. In other words the EDM model has no knowledge that there is a mapping and an ORM engine at all, and can be used or re-used without the presence of the mapping and ORM engine. This is an essential pattern for working in a remote/disconnected scenario where a service is responsible for manipulating and persisting data (and thus is using the mapping layer and ORM engine), and the client solely use the data provided by the services, through the form of EDM objects, with no knowledge on how and where the data is stored. Please note that there is nothing conceptually new here and this is the basis of most existing ORM tools of the market, adapted to the Microsoft terminology.

So now that the introduction is made, here comes the main subject of this post.

The Microsoft Entity Framework entering the ORM tools market

For what I've understood from the .Net Rocks podcast, Microsoft could be having more interest on the EDM model than on the ORM engine. You could imagine that this is because they did not manage to catch-up with the existing ORM tools maturity or for a more strategic reason, but that's out of subject here.

Fact is they are many ORM vendors, doing and mastering this job for years and being very mature. The EDM specifically could be on its side more interesting for MS because it is platform oriented and could become a new Microsoft standard, ideal for making applications inter-communicate though this EDM full object oriented, strongly typed object model. Let be clear: the Entity Framework is not a platform, however the Entity Framework and more specifically the EDM model can become an important glue element of the Microsoft platform, becoming the de-facto standard for inter-connecting Microsoft applications and services and any third party application wanting to leverage the EDM opportunity and offer a communication interface to the outer world though EDM entities.

  

A new business opportunity for third party vendors?

What I would extrapolate from Stephen Forte talk in the podcast (and that engages only me) is that there is an opportunity for ORM vendors to build on top of EDM and provide advanced and efficient ORM engine core functionalities: a third party ORM engine replacing the Microsoft one, manipulating and persisting data all through EDM entities. There is also an opportunity for Microsoft Partners to build tools or designers that could assist creating a EDM model and map it to the database + making use of an ORM engine (being the Microsoft one or any other "EDM Compliant" ORM engine). In this scenario, third parties ORM vendors would be using the EDM as the data model used to manipulate, transfer data, between applications and services. Another clarification: the EDM is indeed not a new, revolutionary concept, and every ORM tool in the market is including such a concept: the entities model is indeed the actual object oriented side of the object/relation mapping layer and is an essential part of any ORM tool.

    

Closer look on the EDM on a broader picture

The EDM model becomes a data contract in the WCF sense, making easier and efficient to exchange data between systems, regardless of where the underlying data is stored and whether or not we are using ORM engine to concretely retrieve these EDM entities. For instance you could most probably use and EDM model to represent data in an inter-application communication scenario and not link to any database in the back-end. The EDM and the Entity Framework ORM engine are loose-coupled and you can use one or the other independently. Well at least you can use the EDM independently from the EF ORM engine, I am not too sure of the opposite.

And the EDM is even more than just a data contract: it comes with tools and will probably come with more. One is the Entity SQL, based on SQL syntax oriented toward object queries on the EDM model. And again this is nothing new: for instance Llblgen provides a full framework around its entity model, adding extra functionalities as compared to a basic object model or to any model you could build from scratch by hand, in a reasonable amount of time.

So back to the point: using the EDM as a data contract, it removes the need for applying a DTO pattern for interoperability between systems and especially when exposing services and wanting to make your clients unaware and decoupled from your inner implementation and especially from the ORM tool you are using. Well to be more precise: if the EDM becomes a "Microsoft standard", any one agreeing on that could consume a service delivering EDM entities. But you should be aware that the interoperability will anyway not be complete as using EDM will probably tight you to the Microsoft platform and "Microsoft Entity Framework aware" clients. I will write another post about the DTO pattern in a SOA world and when to use or not use it. DTO is yet another mapping layer and one can wonder why after mapping from the relational world to the object world we still need another translation. My recommendation would be that one should not introduce another mapping layer is there is no absolute need for it. You might need it or might absolutely want to forget about it, depending on the scenario you're in, there is no single solution to all problems.

So is the Entity Data Model much bigger than a just an ORM?

So Yes, for what the EDM is concerned and what you can hear in the podcast as being "more than just ORM", you can achieve the same result with other ORM tool and I am actually doing it on my projects with the LLBLGen Pro generated entities model layer. What however the EDM might provide as a plus in the future is integration in the whole MS suite and become the missing glue link to seamlessly connect systems though the strongly type, object oriented EDM data model. Well let´s be more precise: the missing link in systems based on Microsoft tools and technology.   

So to insist on this point, I can understand Frans Bourma when he is reacting strongly on the sentence "The EF is much more than an ORM" because no, there is no magic and what MS say they can do with the entity framework and the EDM could be achieved with another ORM tool and its own entity model. It is just after all a matter of convention and all actors standardizing on the same framework (and for instance here the EDM part of entity framework). But Yes MS can bring something more with the EDM that goes beyond the ORM aspect, just by the fact that they can push for using the EDM as the data contract layer for all of their products. They can do it because they own the product! And this is what makes the idea powerful: for instance when a company start investing in Microsoft, it usually spread the whole SI and range of applications: start installing MS servers and desktops windows OS and you end up using Exchange server and SharePoint server and office and more and more, rather that competing products. And why? Just because all integrate so well: it is all coming for a single vendor who actually makes the integration of all its range of applications as easy and straight-forward as possible! Same thing can happen with the EDM on a more technical, low level I must say, but following the same driving idea.

      

To conclude …   

To conclude, what I can foresee is that what will bring most to Microsoft strategically speaking is the EDM layer of the Entity Framework. Because this it becomes part of the communication layer, behind the data container for interconnecting the various MS applications. Then how the ORM engine is internally implemented is not of so much importance. And I guess they will be more than happy at Microsoft if a third party ORM engine could be used, and build on top of the EDM and proving very efficient. It will just go in their direction of promoting the entity framework and especially the EDM layer and through the EDM the maturity of their platform and catalogue of products and they will be in a position to market and sell how easy interconnecting them is.

Introducing my favorite ORM tool: Llblgen

In a previous post I introduced the concept of ORM tools and their high importance in software development projects. Today I want to talk quickly about my favourite ORM tool: LlblgenPro, and explain why it has been one of the best tool I have had the chance to use for quite a long time.

 

Have you ever been really amazed by reading the marketing material for a product and think: hey this is what I need! Looks like it's doing everything I want/need plus even more things I would not have thought of first, let's get it!

You would then go and download the trial, with the grand hope that you will be simply satisfied, and not once more disappointed by an advertising façade, selling why the product is the best of the world.

So here you are, trial downloaded, and first eliminatory test is how easy or hard the product is to install and first use. Once the first, supposed easy test is pass, then you start using the product. If you are lucky, the first tries are satisfactory and you still find the product very attractive and are ready to go and celebrate this with your friends. If you are really that lucky and don't live in the real world, you never discover any problem with your newly purchase product and live happily ever after.

However sad but true, more common scenario is that you discover one by one, sooner or later, the flaws and problems and bugs that goes with the product. And interestingly or not, this was not sold on the advert. But the thing is that it comes in a bundle: you get the great stuff and the bugs and problems all in one for the same price, how great is that?!

Unfortunately this happens quite a lot, it is very common to have to make compromises and use a tool because it has great features and you need it, accepting all the bugs, design and functional flaws, perhaps lack of flexibility that come with it. Well you still can contact this great support team: they say that they are available to solve your problems, so wanna call them? Ok right: one doesn't loose anything to try, except perhaps patience and hair…

Ok all is not so bad. You have products to absolutely avoid, and products that are better and being a pleasure to use despite of some drawbacks. And you have few tools that are absolutely phenomenal, and Llblgen Pro is IMHO one of these.

I have already used Llblgen Pro on several big size projects for several of my clients. For information I am always using the adapter flavour of the templates and runtime as I am working on large distributed tiered-architectures and need the disconnected pattern for communicating between tiers, throw web services.

So I must say I keep enjoying the fact that I really love this product, and cannot find anything significant to complain about, which is very uncommon. And Frans, where are the bugs? Pretty very well hidden I must say, was looking for them but apparently didn't get any luck: I find the generated code and runtime extremely stable and robust.

Ok what else can I say? Well, installing the product is very easy and nothing to report, using the designer is pretty straightforward too, and using the generated code made very easy too: the generated projects are including all the proper references, are compiling perfectly fine, and you then just need to reference the generated project and runtime support libraries to be ready to map to your relational data in a full object oriented way.

Even more important: re-generating your projects after changes to the database, on existing entities or creating new one is extremely stable and pain-free too. May you add items to your projects, add your custom code, all is re-generated taking very good care of the work you have done before.

Ok you will tell me that changing the database has impacts on all your code, not just you data access and business entities layers. Well, ok, but don't forget we live in a full strongly typed object oriented code now. So may you have broken your code after update of the database and regeneration of the business entities layer, well, you will know it at compilation time and fix it very quickly!

Another very important point for me is the quality of the generated code and of the runtime source code. For such a technical tool one expect very high quality of the code base, and you get it with Lllblgen. When using the generated code or runtime libraries, you never get the impression of a poor design or dirty kind of object oriented, having design inconsistency or suspicious quality issues. No, this is only great clean object oriented code, handsomely designed, I just LOVE IT

What else can I say? Well the development efforts are pretty well sustain in that Llblgen Pro keeps constantly up to date with the newest versions of Microsoft .NET framework when they get shipped. And much better than this I must say: Llblgen Pro really makes the most of new features of the .NET Framework very fast. Example is the very early support of Link (though Link to Llblgen) and Dynamic Data.

Another point is the extra features that are provided out of the box for you, for free. Examples are support for auditing, and security (authorizations). So you have access to points of extensibility for your application just by using the tool. Let's see what new exciting features we get in the future.

What else? Oh yes, support. Llblgen Pro has got the most amazing support service through its forums. You get answers extremely fast to your problems and issues and bug fixes integrated quite fast in subsequent releases. And it is working, they really are there to help you, it is not only sugar to make you buy the product!

I see one bad thing though about the tool…. And it is its name!! Just still haven't really managed to adopt it J Well who cares would you tell me?

Just one thing to be aware of: I am telling a lot of great thing about Llblgen Pro, just be aware that you will have to go through a learning curve before smoothly master the framework. However it is the same with any framework and believe me: it really is worth it. And with the addition of Link to Llblgen, developers knowing Link will have their opportunity to leverage their knowledge and shorten the time needed to master the tool.

Ok enough for now, you can just go to their website to get more information on the tool. And I will write more posts on scenarios in which I am using Llblgen Pro. So go, and try it! And NO, I don't have any shares on the product, just like to tell when there is a tool that deserve attention. And I will dedicate a section in this blog, dedicated to registering interesting and useful tools and framework, as it is one part of the whole "Building Business Oriented Software Solutions" concept.

You can also read my post about Object Relational Mapping (ORM) overview

Object-Relational Mapping (ORM) - Overview

Today's talk is about object-relational mapping, what it is, why it is an essential part of software applications. I will then specifically talk (in a later post) about the ORM framework/tool I am regularly using on my projects and provide consultancy on to my clients: LlblgenPro

I don't pretend here to be an expert on ORM tools inner details and design complexity, I am posting here as an ORM tool user and express my return on experience on this matter.

     

It's all about data

The vast majority of software applications are dealing with data and data is the core business value of these applications. May you lose or corrupt this data and you are out of business. In some cases, data is an even more critical asset that also needs to be protected against theft and usage by malveillant people.

    

Data is relational

In the vast majority of cases data is stored in relational database systems. Reason for this is first historical: database systems have been developed over years, and evolved to very performing, scalable, reliable, widely used systems. For such a critical asset of any business, this is not something that will change easily, even though object databases have emerged and are used is some cases. Secondly, the relational representation of data along with the commonly used query language (SQL) are well suited to querying and consolidating large amount of data in a concurrent multi-user mode, computing aggregate information, insuring transactional operations and enforcing integrity of data.

     

Code is object oriented

Object oriented development is now widely accepted and used for its well-known benefits that we will not detail here. Through object oriented design and techniques, code is made a close abstraction of real world and business domain concepts. Many design patterns have been identified that help solve common design problems and leverage the power of object oriented design to build robust, flexible solutions.

    

Impedance mismatch and data access layer

Now we can spot the problem/challenge and the mismatch that exists between application code that is object oriented and need to work on data represented and stored in a relational format. And from a separation of concern point of view we want this mismatch to be encapsulated and abstracted from both the database layer and the application code layer. It is indeed a best practice from a software architecture point of view to separate concerns and loose couple the various aspects of a software project (more on this in a future post). In our specific case we want to have an independent relational database layer created, maintained and optimized by a database specialist who does not need to be concerned at all by the way his data will be accessed and used; and at the same time ensure that application developers can use their best creativity and knowledge on how to efficiently build flexible, robust and maintainable application code in an object oriented fashion without having to deal with the technical intricacy of converting back and forth relational data to objects. And this is where we introduce the data access layer, which purpose is to encapsulate all the tricky details and logic needed to perform the bi-directional transformation between relational data and object oriented application data. From now on, we will name object oriented application data after the denomination of business objects or entities.

          

OK, so can't I develop my own data access layer?     

As developers, most of us have one day developed our own data access layer, taking care of communicating with the database, retrieving and exposing data to the business layers and performing the common CRUD (create/update/delete) actions against the database. After all, this at first does not seem like such a big deal does it? But it very quickly appears that writing proper, efficient and robust data access code is really a hell of a work. For instance how about managing primary keys / foreign keys relationship? How about dealing with strongly typed data rather than simple raw data tables? What about server side pagination? What about complex retrieval of data across multiple tables?

Other fact is that the data access code is highly technical plumbing code that is good candidate to be made generic and thus legitimate the use of a common set of tool, framework and runtime libraries to perform the work for us and concentrate on the core business implementation. So here we are: after writing one's own data access layer on a real, reasonably sized project, one will quite fast conclude that leveraging an existing ORM tool is relevant!

     

So tell me about ORM tools!

One important point to note is that a data access layer does not necessarily have to be ORM layer; in other words, a basic data access layer that only provide encapsulation of the database connections, database commands building and requesting data that is then used by the business layer (probably as ADO.NET datasets) is not a proper ORM layer. The ORM layer goes one step further than a basic data access layer by providing (or connecting to, through a mapping configuration) a business entity model layer on top of the actual data access code and most probably advanced functionalities that can include various interesting stuff like object oriented query language (that could now be exposed through the Microsoft Link extension to .NET languages), web service and distributed application support, auditing, database schema change management, multi-databases, support for data-binding etc.

Using an ORM can therefore save an awful lot of work and give you for free many extra functionality that you can potentially use to improve or extend your applications. The only matter then is to select one ORM tool from the many existing.     

You can see from what exists that ORM tools can be of different nature, you will find:

1. Simple entity objects and data access code generator (which I believe are not really fully fledged and powerful ORM tools). Example could be Code Smith templates.

2. Entity object layer code generator + runtime libraries for data access. Example of such a tool is LlblgenPro.

3. Mapping configuration to connect an existing data store to an existing entity object layer + runtime libraries for data access. Example of such a tool is NHibernate

 

Please note that an ORM tool can also be data centric, model centric or both. A data centric ORM tool will need an existing database schema and generate its entity business model from it; a model centric ORM tool will be able to generate a database from an object entity model. And hybrids ORM tool will be able to connect an existing entity object model and existing database through a configuration mapping file. 

I don't personally find the model centric approach and generation of database relevant. The database should IMHO be designed separately. There are techniques to design efficient well performing databases and I don't believe an automatic generation of the schema is appropriate, except maybe for quick proof of concept prototype or very low load applications. Designing an appropriate database schema is too much dependant on the application business domain and the usage that will be done of it, and the load that will be supported by the application.  

I personally find the data centric approach being the best compromise for most applications. The constraint is that your business model should then be quite close to your data model, which I believe in most business application is the case. It's maybe not the case for warehouse databases designed and optimized for reporting purposes, but there are other techniques than business application development to manage their data and more a appropriate scenario would then to use a specialized data reporting server application (like Microsoft SQL Reporting Services). Another case for which the entity data model might not fit the database schema could be very high load databases for which the database schema has been deeply optimized for performance.

I can see too ways to overcome the constraint of having close entity model and database schema and allowing some divergence between the models. One is to use a hybrid ORM tools allowing connection though mapping configuration between an entity object model and a database schema. However what I don't like about this solution is that you have to write your own entity data model where a data centric ORM tool could automatically generate it for you. Other option is to use a data centric ORM tool and add to the database schema database views that could encapsulate and hide for you the differences between your models. Well of course then you have to handle updateable view or use triggers to manage write operations on the view but hey, that would be the subject of another post!

  

So you now know my preferences. Of course every project has its own specificities and constraints and the choice of an ORM strategy must be revised for every new project. But I always try to save myself the most development work I could, and therefore anytime I can I go for a data centric ORM, because of its ability to generate for you both the entity object model and the actual data access layer, saving you a huge development effort! And my favorite ORM tool, of which I will talk about in a future post, is LlblgenPro.