In working on my upcoming EF 4.3 Migrations video for Pluralsight, I wanted to work out how to use this with an existing database where I plan to add new types and therefore want migrations to not just use this database, but migrate it as well. Problem solved*, but then I tried to use it in a production application and found an easy-to-fix problem. So…I thought I’d share the process in a blog post while it’s on my mind.

The Purpose of the Migration-History Table

When you let EF 4.3 code first create a database for you, it inserts a Migration-History table into the new database. It’s hidden in System Tables. This is equivalent (in a way) to the EdmMetadata table you would have gotten with EF 4.1 & 4.2.

The table is used by two processes (maybe more that I haven’t encountered yet).

When you run the app using automatic migrations, if it’s doing any type of data initialization, it will check that Migration-History table to see if the database needs to be updated.

If you are using code-based migrations, Update-Database will also look at the Migration-History table.

If there’s no migration-history table (which there won’t be in an existing database), the model won’t be verified against the database with automatic migrations.

So you need to get that table into your existing database.

That table is created using the most recent migration file in your data layer.

Getting a Migration-History table into an existing database

If you’re starting with a new app, you won’t have that either.

So

Step 1) In Package Manager Console, execute “add-migration initial”.

This will force code first to create a migration class (which I chose to name “initial”) based on the code first model. It will have migration code to create all of the necessary database tables (based on what it’s found in the model) with their columns, keys and constraints.

But your database already exists! If you were to try to execute that migration and you have entities that map to those existing tables, EF will try to create the tables and tell you that the tables already exist.

Step 2) remove all code that duplicates existing tables from inside the Up and Down override methods in the initial migration. But, leave the methods there.

This is a little tricky. I find that it’s safer to do this when there’s nothing in my model yet that would required a database modification. That way it’s all clean. I’ve done this when I had one new class and then I had to start over again to get the true “initial database” and then start making any mods I want to my model.

Now, code first has a migration that contains no database modification code. That represents how things are at the beginning of EF 4.3 migrations involvement in your app.

Step 3) run Update-Database

This will do two things. It will execute the Up method in the initial migration, which has no effect on the database. And it will create the Migration-History table in your database based on the current state of the model. That way, if you are using automatic migrations, next time Code First does db initialization, it will compare the current model to the one stored in the migration-history to determine if the database needs to be migrated/updated.

Now you can evolve your app and let code first fix up your development database as needed.

The problem I encountered? A SQL Server 2000 Database

I was doing this with an existing client database (not production, but a copy for development … just sayin’  )

When running Update-Database I got the following error:

Conversion failed when converting date and/or time from character string.

Remember I’m not really updating my database. This was when code first was trying to create the Migration-History table.

It is an old database that has been recently updated to SS2008. Even though I’m using SQL Server 2008R2, that database was still set as a SQL Server 2000 database and was not happy with the format of the date field in the INSERT command:

INSERT INTO [__MigrationHistory] 
 ([MigrationId], [CreatedOn], [Model], [ProductVersion])
 VALUES ('201202161546124_initial', '2012-02-16T15:55:56.252Z', 
          [big-ass hash you don’t need to see], '4.3.0')

Luckily, I was able to just update my database version to 2008  and the insert command succeeded with no problem.

*still finding some more interesting tasks I have to do. For example, as I ensure that my model mappings reflect the database, I have to rewrite my initial migration and delete the migration-history table (there might be a scaffold command for that but I’m doing it in SSMS) and call update-database again.

You may already be familiar with the fact that ASP.NET MVC 3 tooling includes a scaffolding option that let’s you point to a domain class and automatically build a Controller, a set of Views and if it does not exist yet, an Entity Framework DbContext class expose that class. The controller CRUD code uses the DbContext to perform it’s operations. Scott’s Guthrie & Hanselman both introduced us to this in blog posts last spring and I’ve certainly demo’d it way too many times.

This is handy but it puts all of the data access code into the controller. In other words, the controller works directly with the DbContext to perform queries & updates (etc).

If you have been paying attention, you may also be aware of the MVCScaffolding NuGet package that adds to these capabilities.

With this installed there are two new templates added to the MVC Add Controller wizard, one which adds a simple repository to the mix.

MVCScaffolding relies on another NuGet package, T4Scaffolding which contains many of the scaffolding templates used by MVCScaffolding.

I was working on the back end of a new app that may or may not have MVC as it’s front end UI. I had created my domain classes and the next step was to create a DbContext to wrap them as well as repositories. I didn’t want to code those by hand. I wanted that MVC tooling to get some of the grunt work out of the way for me. A little reading helped me learn that I could use the T4Scaffolding directly to get the same DbContext & Repository creation without being part of an MVC app (i.e. no controllers or views created).

Here’s how I worked it out.

In my solution where I already had my DomainClasses project, I added a new project, Data Layer.

I then installed Entity Framework via NuGet into this project:

Notice that NuGet automatically picks up the the latest version of EF, 4.3. I could have used the Package Manager Library UI to install EF, but since I had to the rest in the console window, I’m just doing all of these tasks in the console window.

Next, I install T4Scaffolding.

Important Pointers!

• Currently, T4 Scaffolding will install EF 4.1 if you don’t already have EF 4.1 or greater in your project. That’s why I installed EF first, so I can be sure to use the newest version.
• In the package manager console window, I have my default project pointing to DataLayer. I often forget to specify the project and install NuGet packages into the wrong project.

With the package installed, I can now start working towards letting it build out the context & repository for me.

First I need to reference DomainClasses from the DataLayer project

Then I build the solution so that DataLayer can see the DomainClasses for the next step.

Now in the Package Manager Console Window, I’ll use the command to build the repository code. The T4 template that builds the context & repo is called repository. The command is scaffold. I want to run scaffold using the repository template and base the output on my Alias class. I must use Alias’ fully qualified name so that it can be found.

The command is

scaffold repository DomainClasses.Alias

However, I want to specify the name of my context. The default will be DataLayerContext (using the project name). I want it to be TwitterContext.

scaffold repository DomainClasses.Alias -DbContextType:TwitterContext

The result is that a new context & repository class get created in the target project (DataLayer).

There’s some default behavior that is not exactly to my liking and I can use additional parameters as well as modify the T4 template (here’s a post from Steve Sanderson as part of the MVC Scaffolding series that will help with that) but for now I’m happy to just move files around, which I’ll do shortly.

Here’s the context class.

    public class TwitterContext : DbContext
    {
      public DbSet<DomainClasses.Alias> Aliases { get; set; }
    }

Now I’ll add another class from my domain model to the mix: Tweet.

Notice that since TwitterContext already existed, we don’t get a second context class, the template alters the existing one:

    public class TwitterContext : DbContext
    {
      public DbSet<DomainClasses.Alias> Aliases { get; set; }
      public DbSet<DomainClasses.Tweet> Tweets { get; set; }
    }

Finally, I’ll just move things around so they are more to my liking:

I’ve created a Repositories project that has references to EF4.3, DataLayer and DomainClasses.

Now I can use the context & repos as they are or take the generated code (which has saved me a lot of time) and tweak it to my needs. But this has taken a lot of the repetitive typing away from me and started me on a path of success. Happiness.

Thanks to Steve Sanderson & others who worked on this tooling!

(Oh and if you could modify the template to pull in “latest version of EntityFramework.dll”, that would be handy.)

My newest course is live on Pluralsight.com. This one is all about performing data layer validations in EF 4.1 (4.2, 4.3) using the Validation API that’s part of the DbContext.

Data Layer Validation with Entity Framework 4.1+
Working with the DbContext Validation API
(total 1 hr 54 m)

• Module 1: Data Layer Validation with Entity Framework DbContext
• Module 2: Customizing Entity Framework Data Layer Validation
• Module 3: Integrating Entity Framework Validation with MVC and WCF Data Services

Here’s the entire list of EF courses I’ve done for Pluralsight (so far ). I’ve sorted them in the order you might want to watch them in.

Entity Framework (7 courses)

Entity Framework 4.0 By Example
Beginner
[02:18:50]
27 Aug 2010

Entity Framework and Data Models
Intermediate
[01:31:38]
5 Nov 2010

Designer Supported EDM Customization
Intermediate
[02:05:13]
6 Jan 2011

Querying the Entity Framework
Intermediate
[01:22:37]
28 Feb 2011

Entity Framework 4.1 – Code First
Intermediate
[01:58:17]
18 Jun 2011

Entity Framework 4.1 – DbContext Data Access
Intermediate
[01:21:19]
27 Sep 2011

Data Layer Validation with Entity Framework 4.1+
Intermediate
[01:53:49]
8 Feb 2012

Coming soon: Entity Framework 4.3: Code First Migrations and other changes

A friendly reminder about the UserVoice site where you can suggest and vote on features you’d like to see in Entity Framework (http://data.uservoice.com/forums/72025-ado-net-entity-framework-ef-feature-suggestions) or in WCF Data Services (http://data.uservoice.com).

The EF team is wrapping up work on changes to the core EF APIs that will be part of the .NET 4.5 release. We’ve had access to the preview of these new features since June 2011. Here’s an MSDN Magazine article reviewing some of the most prominent features that you’ll find in there: http://msdn.microsoft.com/en-us/magazine/hh394151.aspx.

But EF continues to evolve. The team pays close attention to the suggestions developers make (and vote on) on user voice. In fact, it’s a bit of a popularity contest. If one feature suggestion has hundreds of votes, it will get a lot more attention from the team than a suggestion with only 10 votes.

Here’s a look at hot suggestions.

Top 10 Suggestions Overall

Improved SQL Generation (status=Started) 1204 votes

Batch CUD support (status=Under Review) 981 votes

EF Support for Second Level Cache (status=Under Review) 535 votes

Entity Designer: Speed up & Optimize for using with ~200+ entities 453 votes

Support for multiple databases (status=Under Review) 441 votes

Designer Support GUID as Entity Key  429 votes

Schema Migration a la Rails Migrations (status =Started (Code First Migrations)) 406 votes

TPT (Table-Per-Type) Inheritance Performance (coming in .NET 4.5) 369 votes

Allow filtering for Include extension method (status=Under Review) 319 votes

Allow default date = NOW and provider.MinDate in CSDL (my suggestion ) 216 votes
(this isn’t as much of a problem now with code first, but for EDMX it’s still an issue)

There are currently 276 overall. Some have been completed. Some may be duplicates. But if there are features you care about getting into EF, take a look at the list and vote and *promote* the ones you care about.

Oracle has finally released it’s version of ODP.NET that supports Entity Framework for use in production environments.

This is great news for many developers! 

And they report that this release supports EF 4.1 and EF 4.2. That should mean Code First and DbContext but thanks to Frans Bouma sending me the limitations doc (that is part of the installation ) we can see that they aren’t! Too bad.

7. ODP.NET 11.2.0.3 does not support Code First nor the DbContext APIs.

So, that means “not really EF 4.1 and EF 4.2 support” in my opinion. But they told us already that Code First & DbContext weren’t going to make it in. So it was just the momentary surprise of misunderstanding their meaning about EF4.1 & 4.2 support that deflated me a bit. 😉

Howver…I imagine Code First and DbContext are still leading edge for the types of big corporations that use Oracle. The core EF 4 support is a big deal.

I’ll follow up with Oracle for clarification.

More from Oracle (including downloads) here: http://www.oracle.com/technetwork/topics/dotnet/whatsnew/index.html

Note: They’ve updated the what’s new page to say "Code First is not supported." That’s helpful. Be sure that even thought it’s not mentioned there, EF 4.1/4.2 DbContext API is not supported either.

I received this pair of questions about EF in email and thought I would share the questions and my reply.

Q:

I’ve read some articles about Code First, I’ve tried to do some samples and I’m going to read you last book.

I’m surprised about 2 issues. The first is that it’s possible to write native SQL code (with context.Database.ExecuteSqlCommand and context.Database.SqlQuery methods) and the second is that I’ve not been able to use EntitySQL or Query Builder Methods.

Now, the idea of EF was create a model, operate with objects and use a common language (EntitSql) to interact with "some" databases. But now? We are coming back? Yes, we can use Linq2Entities and SaveChanges but I don’t understand the philosophy of these opened doors upon the database.

I’m wrong?

What’s your idea?

My A:

Dbcontext doesn’t replace ObjectContext. It just makes it easier for devs to access the most common things.

You can still use ObjectContext. You can even use DbContext and access ObjectContext functions when you need to.

Entity SQL was created before the data team knew about LINQ. You can express most queries with LINQ and it works with any EF-compliant database provider, just as ESQL does.

The direct database access command features are a "backdoor" that devs asked for. It should be used only on rare occasions when you need to access something in the database that is not part of the model.

Entity Framework’s Include method which eager loads related data in a query does not allow sorting and filtering. People ask for this feature frequently.

There is a way around the problem,  I wrote about it in  the June Data Points column in MSDN Magazine (Loading Related Data: http://msdn.microsoft.com/en-us/magazine/hh205756.aspx) but since I just saw another comment on twitter about this, I thought I would stop working on Chapter 7 of the DbContext book that is already delayed and write a quick blog post about using a projection instead.

One caveat…you cannot do this if you have a long running context and you want the objects to remain attached to the context. If there are other related entities in the context, the context will always give them to you. More on this at the end of the post.

Okay, today’s model will be based on a roads and trees on that road. I need some type of guide because we’ve been socked in with fog for two days and I can barely see the trees across my road.

We’ll have two classes (and no I’m not trying to rub in the fact that we don’t have geo support yet, but it’s coming in .NET 4.5.)

public class Tree
  {
    public int Id { get; set; }
    public string Description { get; set; }
    public decimal Lat { get; set; }
    public decimal Long { get; set; }
    public int RoadId { get; set; }
  }
  public class Road
  {
    public Road()
    {
      Trees = new List<Tree>();
    }
    public int Id { get; set; }
    public string Name { get; set; }
    public string Town { get; set; }
    public List<Tree> Trees { get; set; }
  }

The way loading works in Entity Framework is an all or nothing scenario with one exception.

The DbContext lets you do a filter in an explicit load (i.e. after the root is already in memory). Here’s a passing test which loads only maple trees and checks to see that the count of non-maple trees is NOT greater than 0.

(MyInitializer seeds the database with one road that has three related Trees (a maple, a maple and a pine)).

    [TestMethod()]
    public void CanFilterOnExplicitLoad()
    {
      Database.SetInitializer(new MyInitializer());
      var context = new DataAccess();
      var road = context.Roads.FirstOrDefault();
      context.Entry(road).Collection(r => r.Trees)
        .Query().Where(t => t.Description.Contains("Maple"))
        .Load();
      Assert.IsFalse(context.Trees.Local
                            .Where(t=>!t.Description.Contains("Maple"))
                            .Count()>0);

    }

But that’s beside the point, since the question is about eager loading.

You can’t filter when eager loading with Include.

But you can get the results you want eagerly if you project.

  [TestMethod()]
    public void CanFilterOnProjection()
    {
      Database.SetInitializer(new MyInitializer());
      var context = new DataAccess();
      var road = context.Roads
        .Select(r=>new{
          r,
          r.Trees.Where(t=>t.Description.Contains("Maple")})
        .FirstOrDefault();
      Assert.IsFalse(context.Trees.Local
                            .Where(t => !t.Description.Contains("Maple"))
                            .Count() > 0);
    }
  }

That’s nice but whine whine whine, I returned an anonymous type.

If you use a repository, you can hide that.

  public class DumbRepositoryButGoodEnoughForThisDemo
  {
    DataAccess _context=new DataAccess();

    public List<Road> GetRoadsWithFilteredTrees(string treeFilter)
    {
      var roadAndTrees = _context.Roads
       .Select(r=>new{
          Road=r,
          Trees=r.Trees.Where(t=>t.Description.Contains("Maple"))})
        .ToList();
      return roadAndTrees.Select(rAt=>rAt.Road).ToList();
    }
  }

When I return the list of roads from the projection, the Trees will be attached thanks to the context recognizing the relationships.

Here’s another test that does pass:

   [TestMethod()]
    public void RepositoryFilteredRoadsReturnsRoadWithTrees()
    {
      Database.SetInitializer(new MyInitializer());
      var rep = new DumbRepositoryButGoodEnoughForThisDemo();
      var roads = rep.GetRoadsWithFilteredTrees("Maple");
      Assert.IsTrue(roads.FirstOrDefault().Trees.Any());
      Assert.IsFalse(roads.FirstOrDefault()
                          .Trees
                          .Where(t => !t.Description.Contains("Maple"))
                          .Count() > 0);
      
    }

And a screenshot as further proof:

Scenario When This May Not Work As Expected

As Brian points out in the comments, there is one BIG draw back that is also a problem with the filtered explicit load. If there are already related entities tracked by the context, they will automatically be attached to any related entity in the context. That means if the Pine tree was already being tracked then as long as the Road is attached to the context, it will see ALL of the related trees in the context, including the Pine that was already there.

If you are using a pattern where you have a short-lived context that is instantiated just to execute the query, then it’s not a problem. Most of my architectures are like this. But if you are writing a Windows Form or WPF form and have a context that hangs around, you could run into this problem.

You’ll never avoid the problem when the entities are attached to the context.

But here’s a twist on the repository method that will return disconnected objects from a context that is managing multiple object.

   public List<Road> GetDisconnectedRoadsWithFilteredTrees(string treeFilter)
    {
      var roadAndTrees = _context.Roads.AsNoTracking()
       .Select(r => new
       {
         Road = r,
         Trees = r.Trees.Where(t => t.Description.Contains("Maple"))
       })
        .ToList();

      var rt = new List<Road>();
      foreach (var r in roadAndTrees)
      {
        r.Road.Trees.AddRange(r.Trees);
      }
      return roadAndTrees.Select(rAt => rAt.Road).ToList();
    }