Category Archives: Data Access

What’s Coming Up in EFCore’s Near Future

Data Access, ,

EF Core 1.0.0 was released June 27 along with ASP.NET Core.

Even before that date, more goodies were added and fixes were made that were not able to get into that package. For example, we’ve got DbSet.Find back now in the nightly builds and a fix for a big performance problem [that affected a narrow set of use cases which is probably why it went undetected for so long] which related to the asynchronous eager loading (Include). It was fixed within a few days of the issue being created.

There is a patch (1.0.1) coming hopefully in early August. (From dotnet blog linked below: “There is no scheduled date for this patch update but early August is likely.” ) But that is only a patch to fix important bugs. (Maybe the async include performance fix will be in there but Rowan Miller explicitly said “Find” is in 1.1, not 1.0.1.)

2016-07-25_13-51-30

1.1 is feature release and is planned for possibly Q4 2016. This is the one where we go from project.json back to csproj/msbuild.  From the mid-July dotnet blog post titled .NET Core Roadmap , below is the list of the important features that will come to EF Core at that time.

If you don’t see what you are looking for I have two suggestions:

  1. Look at the EF Core roadmap on its Github repository. You can find it at bit.ly/efcoreroadmap.
  2. Search the github repository. For example, here’s a search for those of you looking for TPT conversations: 

Entity Framework Core Minor Update (1.1?)
Q4 2016 / Q1 2017

  • Azure
    • Transient fault handling (resiliency)
  • Mapping
    • Custom type conversions
    • Complex types (value objects)
    • Entity entry APIs
  • Update pipeline
    • CUD stored procedures
    • Better batching (TVPs)
    • Ambient transactions
  • Query
    • Stability, performance.
  • Migrations
    • Seed data
    • Stability
  • Reverse engineer
    • Pluralization
    • VS item template (UX)

 

Video of My .NET on a Mac Demo at DotNetFringe

Data Access, dotNET, Tools, , , , ,

Last week at the awesome DotNetFringe conference in Portland, Oregon, I did a 30 minute demo of building an ASP.NET Web API with Entity Framework using Visual Studio Code on my lovely MacBookPro. So it’s .NET on a mac (coding, debugging and running). It is *that* cross platform.

I also talked about some of the features of ASPNetCore and EFCore. I used other cross platform stuff like JetBrains’ DataGrip IDE for interacting with numerous databases on numerous platforms, PostgreSQL database, xunit for testing and more!

It was a boatload of fun and it’s on YouTube:

The solution I showed in the demo is in my github repository: julielerman/EFCore-ASPNetCore-WebAPI-RTM

Designer Support for EF Core via DevArt

Data Access

Although Microsoft has dropped support for EDMX as it moves to the next generation of Entity Framework, 3rd party providers have promised to continue supporting designer based tooling for EF Core. Since the EF team announced that it wouldn’t support EDMX going forward, I have asked the providers directly and shared their response that yes, they were planning to support EF Core. And I’ve been trying to calm the many developers who currently depend on EDMX and/or the designer and really want to use EF Core rather than sticking with EF6 and the existing tooling.

The two providers that have long had their own designer support for Entity Framework are LLBLGen and DevArt.

DevArt just announced the release of Entity Developer 6.0 which does indeed support EF Core. Within minutes of reading their tweet announcing this, I took it for a spin. Even though I’m mostly using Code First exclusively these days, I know a LOT of developers who have huge investments in EDMX and would love to use EF Core. So definitely wanted to see how this works.

I grabbed the Entity Developer 6.0 Professional Trial (the trial is free, a real license is not). Although Entity Developer has been around for years and I do have clients that use it for working with Oracle, this is the first time I’ve ever tried it out myself.

You do not need a designer to reverse engineer from the database! EF Core can reverse engineer from an existing database into POCOs and a DbContext. (Check out the scaffold feature in my MSDN Magazine article on EF Core Migrations.) The value of this designer with respect to EF Core is a) if you have an existing EDMX and want to continue using it or b) you want to do designer based modeling for EF Core.

Highlights:

  • The most important part of the process is that this designer generates the DbContext class using the mapping and connection details needed by Code First and does so using EF Core APIs and namespaces.
  • You can import an existing EDMX into their designer.
  • You can create a new model via Database First or Model First in their designer.
  • There is a special template for EF Core to generate DbContext and POCO classes from the model.
  • You then use those classes in your application where EF Core as though you were using Code First.
  • You can continue using the designer paradigm just as you’ve been able to do with EDMX. That means you can modify the database and update the model based on those changes, then regenerate your classes. You can tweak the model visually in the designer and regenerate the classes. You would not use Code First Migrations in this case.
  • If you want, you could drop the designer after the first code generation and continue with code first and migrations. In that case, unless you really want to use the designer for this first iteration, you could use a straight database to code first workflow which is provided by EF Core migrations “scaffold” command or keep an eye on ReversePoco.com for it’s EFCore support.

Screenshots of creating a new model from an existing (very simple) database.

Create a new model, selecting EF Core as the model type.

image

Tell it that I want to use an existing database.

image

Choose a database provider (I picked SqlServer)

image

Select objects. I didn’t have a lot in my tiny demo database. You can see I chose a database that I had created earlier via code first…that’s why there’s a MigrationHistory table there.

image

Set up naming rules  Entity Developer has always had this. I just left it at defaults.

image

Define Model Properties. Notice some EF Core specifics on this page. Many to Many and Table Per Type support are disabled because they are not [currently] supported in EF Core. Also shadow properties is a new feature in EF Core and there is an option related to that.

image

Select a template  EF Core is there by default but you can create and save custom templates.

image

 

After this, the model gets generated and displayed in the designer where you can have your way with it.

image

 

Generate the code. This is an menu option under “Model” (or just press F7). This is the part I was sorely interested in of course.

image

When generation is complete, you get a message. It has created a DbContext class and one POCO class for each of my entities.

image

Then I created a new solution in Visual Studio a Domain Classes project and a Data Model project and copied the new classes into the relevant projects. Until I installed EF Core, there were loads of build errors.

image

Install EF Core into the DataModel project. It’s been a while since I used Powershell to do this so I forgot to add “ –pre” to the command. I’m sharing that because I’m sure others will make the same mistake! So don’t feel bad. I do it too! Smile

image

The POCO classes are not very interesting. They’re just POCOs. The context class however, is where you can see that it’s using EFCore. I’ll share the key parts of that class.

EF Core namespaces:

using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata;

EF Core OnConfiguring along with the EF Core DbContextOptionsBuilder and the UseSqlServer extension method.

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(@"Data Source=(localdb)\mssqllocaldb;Initial Catalog=NinjaDomain.DataModel.NinjaContext;Integrated Security=True;Persist Security Info=True;User ID=;Password=");
            CustomizeConfiguration(ref optionsBuilder);
            base.OnConfiguring(optionsBuilder);
        }

Comprehends the new relationship mapping methods in EF Core:

  #region Ninja Navigation properties

  modelBuilder.Entity<Ninja>().HasMany(x => x.NinjaEquipments).WithOne(op => op.Ninja).OnDelete(DeleteBehavior.Cascade).IsRequired(true).HasForeignKey(@"NinjaId");
  modelBuilder.Entity<Ninja>().HasOne(x => x.Clan).WithMany(op => op.Ninjas).OnDelete(DeleteBehavior.Cascade).IsRequired(true).HasForeignKey(@"ClanId");

Enough for me for now

That’s all I did for the time being. I wanted to see the basic workflow. The big difference between EF6 (and earlier) and EF Core with respect to this is that at runtime,  EF6 (etc) would read the XML files that represented the EDMX and figure out the model from that. Even though the EF Designer generates code, that code is only for the sake of your own business logic. At runtime, if EF sees you are using an EDMX to describe your model, it relied on the XML files to learn about mappings to comprehend the model. EF Core *only* knows how to read the DbContext & classes, so it was key for the code generation to build those classes with all of the mapping details (and expressed via the EF Core APIs) that Code First needs to comprehend the model at runtime. That’s what the Entity Developer designer is doing for you. And as you can see from the screenshots, there was more to creating this support than simply modifying the T4 templates.

Updating to RC2: Changes to EFCore, ASPNETCore, PostgreSQL driver & XUnit

Data Access, , , ,

Here’s my EFCore RC2 Demo on Github

The long-awaited 2nd release candidate came out earlier this week (here is the team’s announcement: Announcing ASP.NET Core RC2).

I have been sticking with RC1 and avoiding mucking with the nightly builds as Microsoft evolved this stack towards RC2 because so much was changing. Others were braver such as Shawn Wildermuth, the beardy Shane Boyer and the sad people who have been building products and tools dependent on ASPNET Core.

So as soon as the new bits were released I sat down to finally update the sample application I’ve been using at conferences since March that were still on RC1. Although it’s all cross-platform, I’ve been working on this sample on my MacBook directly in OSX and using only the CoreCLR just to prove to myself that this stuff is truly cross-platform. I’ve even set Mono aside.

I had two tiers of problems to attack: first getting the new CoreCLR (and the CLI that gives us dotnet commands, replacing dnx, dnvm and dnu) onto my Macbook, second was updating my sample which was dependent on a variety of technologies and APIs.

I’ve pushed the updated version of the application to github and renamed it: https://github.com/julielerman/EFCore-RC2-Demo. But I want to share some of the things I did to get this all working. Shawn wrote a great post on updating his app and the aspnet documentation has a doc on updating from RC1 to RC2 as well. I would start with these. I will focus on some of the changes I had to make that either aren’t covered in those posts, aren’t as obvious or just gave me extra heartache.

I would definitely recommend looking at my repository though since it does have working code so if you’re stuck on something and it happens to be something I already did, you might find an example to borrow within my app.

Getting the RC2 CLI onto My MacBook
For most, it should be easy. Just follow the instructions on Microsoft’s .NET Core page (there are different pages for different platforms).

It seemed to work. But I quickly ran into some problems when I was tryingi to convert my code and those led me to a few threads that said I may not have the correct version. There is some question about how much needs to be removed from your system so that you don’t have leftover older bits of CoreCLR hanging around causing conflicts. In fact, a key element of the update is to run an uninstall script (on the instruction page linked to above). There was a pull request to update that script to do a better job of cleaning the old version out before installing the new version. (Since then that PR has been modified and merged and is now what you will get from the instruction page.)

I grabbed the existing pull request version of the script and ran it then re-ran the installer. So began a bunch of lost and frustrating hours because suddently the dotnet command was gone. I kept getting “dotnet command not found” errors. (See next section about that). Keep in mind that I’m still fairly new to OSX so some of my lost time was due to things like being told to go to the “usr” path and my windows brain translating that to users/julielerman. I made the mistake of going to the latter. That wasted a lot of time. But I eventually got it sorted out.

Because I did not have a straight path to success, it is difficult to say what exactly was wrong and what was right. Hopefully the new uninstall script will do the trick. If not I suggest looking at ideas in this thread: dotnet command can’t be found on osx with RC2 bits

Dotnet command not found on OSX?
There are two important takeaways in solving the problem of dotnet not showing up on OSX.
1) For those using zsh as the default commandline instead of bash:
“There is an issue in the way path_helper is working with zsh on OSX. The easiest way for you to get unblocked is to simply symlink the dotnet binary to /usr/local/bin using the following:
ln -s /usr/local/share/dotnet/dotnet /usr/local/bin

This may eventually not be necessary. See this issue for a conversation about making the uninstaller smarter on zsh.

2) Problem with dotnet command on bash
Another user had the same problem but on bash so the symlink was not the answer. Eventually he discovered that some app had installed the following into his ~.bash_profile file:
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/GitHub.app/Contents/Resources/git/bin:/Applications/GitHub.app/Contents/Resources/git/libexec/git-core
He removed it completely and got success although Zlatko K (from the cli team) made this suggestion:

You can actually keep that and just have the following:

export PATH=$PATH:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/GitHub.app/Contents/Resources/git/bin:/Applications/GitHub.app/Contents/Resources/git/libexec/git-core
I don’t know who did that to your .bash_profile but that is the reason why it is happening. And yes, they should not be doing this. They should be adding your path as well.

Updating Visual Studio Code and (yippee) Debugging!

Since I’m doing this on a mac, I don’t have all the benefits of Visual Studio, but Visual Studio Code is pretty awesome. And with it’s limitations, it really forces me to learn my lessons more deeply & memorably, whereas with VS, I can get away with more thanks to all of the extra support. If I was on my regular computers (read: Windows) I would absolutely be using VS, not VSCode). So doing all of this on the Mac gives me some advantages in having to comprehend everythnig I do more clearly. And, it’s fun using the Mac after only using Windows for so long. But I’m not trying to tell anyone they should do all of their .NET coding on a Mac.

So with that out of the way … 🙂  With RC1, VS Code did not have debugging support. So I had to let errors happen and then read the stack output to figure out what was wrong. Again, much harder, but good lessons for me. 🙂 But please do be sure to update VS Code and the C# extension to their latest versions. VS Code will help you create a default launch.json file so that the debugger knows how to launch your app and a tasks.json for the defining the build task that’s specifed in the launch file.

This happens when you debug if there is no launch.json yet. Notice the box to the right of DEBUG and green arrow. It says “<none> ” since no launch file was found. I’m being prompted to create a new one and to select from either Node.js, VS Code EXtension Dev, .NET Core or Chrome. .NET Core is the one you want.

It will create and display the new file in the edit window.

There is one change you need to make to the default which is to point to the dll of your app that gets built by build. I failed at that because I had not noticed a left-over angle bracket from the place holder. 

Here is the original line that you get with the placeholder:

"program": "${workspaceRoot}/bin/Debug/<target-framework>/<project-name.dll>",

The target framework is (currently) netcoreapp1.0. My project name is src because I’m a dope and never changed that folder name. So the dll is in the /src/bin/Debug/netcoreapp1.0/ folder. Therefore the new value is:

"program": "${workspaceRoot}/src/bin/Debug/netcoreapp1.0/src.dll",

Updating the Code

Once I had this sort out, it was time to modify my code. Again, I would start by following through Shawn’s and the teams posts (links above) about the obvious and not so obvious things to hit. This will get things like namespaces covered, an updated main method in startup etc. I will call out changes that were significant to me. But please do look at the commits and history of my github repository for more details.

Wiring up EF and ASPNET Core’s Services for Dependency Injection in Startup.cs

This code looks a lot different. We used to explicitly AddEntityFramework, then Add[dbprovider] then set up the context for injection providing options. Here is the old code

services.AddEntityFramework().AddNpgsql()
.AddDbContext
(options => options.UseNpgsql
(Configuration["Data:PostgreConnection:ConnectionString"]));

With the new code, you just add the DbContext and that signals the obvious dependency on EntityFramework. Similarly, the Use[dbprovider] method triggers the obvious dependency on the provider. Much more succinct. 

services.AddDbContext
(options =>options.UseNpgsql
(Configuration["Data:PostgreConnection:ConnectionString"]));

Changes in my DbContext Class 

I was using an overloaded constructor of DbContext that allowed me to pass in an ASPNetCore service provider. This was to enable a fancy way of letting my integration tests leverage the IoC  services and the dependency injection. The DbContext overload that took in a servicePRovider has disappeared because there is a better way.

So I deleted that constructor of my weathercontext class.

I have some hack code in my context class to ensure that my database table names are “pluralized” since that does not exist yet in EFCore. I read somewhere about at least making the table names align with DbSet names but a) I didn’t see that kick in yet and b) I typically don’t create DbSets for every entity that maps to a database table. So I’ve left the hack code in. You can see it in the WeatherContext class. It just iterates through the entities known by the context and specifies that the relevant talbe should have the name of the entity plus an “s” . I got that from a gist shared by Rowan Miller (EF program manager).

Use of ServiceProvider in an Integration Test

More interestingly, the changes to how to user the service provider allowed for cleaner code in my test class. Note that I have two version of the test class. One is simple where I instantiate the context directly and pass in options specifying to use the InMemory provider. The other is built to take advantage of aspnet core’s IoC/DI services. That’s the one that I’m changing.

Originally in that test class’ constructor I had to create a servicecollection (that ties back to aspnet core services) and to that, specify that I want to use EF and the InMemory database. This is similar to the original code in Startup that I changed above.

_serviceCollection = new ServiceCollection();
_serviceCollection.AddEntityFramework().AddInMemoryDatabase();

The change to this code is similar to the change I made in the startup file:
_services = new ServiceCollection();
_services.AddDbContext(options => options.UseInMemoryDatabase());

(Note that I changed the variable name to _services)

Next in the setup code where I create the context instance and populate it, I was using the services to create a ServiceProvider and passing that into the WeatherContext constructor. That is how the context got the info about the database. I copied this code from the MusicStore sample and am now scratching my head about it because I’m still instantiating the WeatherContext and I still have that options parameter. So it makes no sense. Nevertheless, that code is gone and I’m now using the _services to do their job :

var serviceProvider = _services.BuildServiceProvider();
var serviceScope = serviceProvider.GetRequiredService().CreateScope();
context = serviceScope.ServiceProvider.GetService();

The provider will now to instantiate the context for me and the _services is aware of the InMemory provider. I wonder if I could have written it that way before. But thankfully the fact that the DbContext can no longer even take a ServiceProvider as a parameter so I got forced into the path of writing a better version of this code. 🙂

Speaking of Testing: Changes for xUnit APIs
My tests use xUnit. Brad Wilson and team have updated xUnit to work with RC2. This requires some changes to the project.json used in your test project. I looked at the documentation and focused my gaze on the dependencies section, missing an important detail just above it.

Please do look at the doc “Getting started with xUnit.net (.NET Core / ASP.NET Core)“. While all the info is there, let me just highlight the critical stuff, especially the one I overlooked!

The bit I overlooked is that there’s a new parameter called testRunner and it’s value is, surprise, xunit.

The names of the packages in dependencies are the same but their versions are updated. Here is what I had to change in my project.json for my test project:

"testRunner": "xunit",
"dependencies": {
"xunit": "2.1.0",
"dotnet-test-xunit": "1.0.0-rc2-build10015"
},

Equally important is to update the location of the nuget package. That goes in the new Nuget.config file where you list the package sources.

Here is the line in my Nuget.config that specifies the source where the RC2 version of the xunit packages live:

<add key="xunit" value="https://www.nuget.org/api/v2/" />

Updating Refs to PostgreSQL
When I’m in my app proper (as oppsed to tests), I’m using PostgreSQL. The RC2 package has a different name than RC1. And it, too, has moved to a stable package source.

In project.json I switched out:
“EntityFramework7.Npgsql”: “3.1.0-rc1-3”,

for
“Npgsql.EntityFrameworkCore.PostgreSQL”: “1.0.0-*”,

Note that Microsoft is asking all of the providers to follow this naming pattern using EntityFrameworkCore.

The package source has also moved to a stable location. Here is the relevant listing in my Nuget.config file:

<add key=”NuGet.org” value=”https://api.nuget.org/v3/index.json” />

I had to be sure to get rid of cached packages on my computer. Because I’d been trying and retrying things and not always doing it right, I found myself clearing out the cache where the package lists live on my computer (remember I’m on a mac)

~/.local/share/nuget/v3-cache/ then find the folder that contains the name of the source e.g. “****api.nuget.org.v3”.

Also I had to get rid of the actual packages. Those are in

~/.nuget/packages/[nameofpackage]

Again because I’m an OSX newbie, I’ll clarify that the “~” represents your home folder as a user. I wish I’d had help with this, but everyone presumes you know what that ~ means. Here’s my File system. MacintoshHD is my computer. I’ve done a trick to display the hidden files & folders .. those are the ones that are gray, not black. Ive drilled into Users, then me (julialerman). That’s ~. That’s the starting point. Antyhing after that is in the next column. You can see the .local folder where the . I’ve chopped it off in the screenshot but the .nuget folder is just below.

folders

That should help you be sure you’re getting the right packages.

Dont forget to keep restoring the packages. At some point (when I had more and more of the right things in place, e.g. the launch.json, etc), Visual Studio Code’s prompt to do a restore finally switched over to running the “dotnet restore” command instead of the old “dnu update”.  But for a while I was just running dotnet restore at the command line and you can see the difference between when it’s finding the right packages and when it’s not.

It’s a process

I should remind you that this did not happen in one smooth sitting. Working in the code I had to get the packages to restore without errors. Then I had to modify the code and build and look at the list of errors and then fix them and build again and fix an dbuild and fix and build and fix and build.

Once I got build down to 0 errors, then I could try to run. I was back and forth from running at the command line (dotnet run) and debugging in Visual Studio Code when I needed to really see step by step where an error was coming from.

The Controller Class

I didn’t really have to do anything to the controller class. I had stopped using the special EF change tracking parameter that the team was experimenting with (a parameter of Add/Update/Remove) because I knew it was going away.

I knew that a nice change to MVC in RC2 was that you don’t have to explicitly inherit from the Controller class if you are following one of the conventions that alert .NET that this is a controller class. So I removed it from WeatherController and ValuesController but then added it back in to the first since I didn’t feel like adding Dispose into the class. 🙂

Seeding Data

EFCore does not have a built in way to seed data yet like we’re used to with Code First. For RC1, I borrowed an example from the MusicStore sample for seeding data and modified it to suit my needs. That’s the SeedData class in my repository. This class is called by the web api’s startup and takes advantage of the asp.net services (the IoC stuff) so I can interact with EF and save the seed data into the database. I had to modify my RC1 version to align with the changes I’ve talked about already with respect to how to work with the context “out of band”. So the changes I made here are similar to the changes that I made in the controller class that I mentioned above.  Essentially, I have it use the SErviceProvider to instantiate the WeatherContext for me using the defaults already defined in startup.  Since I showed that change above for the test class, I won’t bother repeating it here. Because this sample is a demo, I use EF’s EnsureDatabaseDeleted and EnsureDatabaseCreated in the seed class to drop and recreate (and reseed) the database every time I run the app.

EF Migrations

My demo also uses migrations so I had to get all of that working again as well. Most of that change is related to the project.json file.

Up through RC1, the migrations commands were found in the EntityFramework.Commands package. Additionally every provider came with a second package with the name .Design appended to it, that you needed to have in order to use the EF commands.

So my RC1 project.json for the app (not the tests) had these in the dependencies:

"EntityFramework7.Npgsql": "3.1.0-rc1-3",
"EntityFramework7.Npgsql.Design": "3.1.0-rc1-5",
"EntityFramework.Commands": "7.0.0-rc1-final",

EntityFramework commands have been wrapped into what is now considered the tooling for CoreCLR.  So it’s package has the name tools now and also all of the tooling is in a different path to being ready. Their verson indicates that. This is the new package name alongside the new package for Postgres that I showed before:

"Npgsql.EntityFrameworkCore.PostgreSQL": "1.0.0-*",
"Microsoft.EntityFrameworkCore.Tools": "1.0.0-preview1-final",

Notice that the Design package is not there. I forgot to add it in and my migrations worked so it turns out that they may not be needed (at least not for what I’ve done so far with migrations).  That was even a surprise to Shay Rojansky who has built the provider. 🙂

However there is another important piece to getting migratons to work and its a big change to project.json. There is a new tools section you must add.  The Kestrel server for IIS integration also has to be added into tools so I’ll show you the entire tools section from my project.json file:

"tools": {
"Microsoft.AspNetCore.Server.IISIntegration.Tools": {
"version": "1.0.0-preview1-final",
"imports": "portable-net45+win8+dnxcore50"
},
"Microsoft.EntityFrameworkCore.Tools": {
"version": "1.0.0-preview1-final",
"imports": [
"portable-net45+win8+dnxcore50",
"portable-net45+win8"
]
}
}

In RC1, we also added in shortcuts for the commands. To use the kestrel server, we could type dnx web at the command line and “web” was a shortcut for the IISIntegration. To use migrations we started with “dnx ef”. EF was a shortcut to the EntityFramework.Commands assembly. Commands no longer exist. Instead we list the tools. But how do we specify the shortcuts? Yes I was wondering. Well for entity framework, its built in. I was able to type “dotnet ef” and get the happy little magic unicorn along with the listing of the commands available for ef migrations.

dotnetef

I’ve written an extensive article on the new migrations commands for EFCore if you want to learn about the commands. That article also includes the powershell versions of the commands that you may be familiar with in Windows e.g. add-migration.

There were more changes, but none that stand out in my memory as much as these. You can see the working code in my repository and look through its history to see how its evolved. Unfortunately isn’t just one commit that brought me from RC1 to RC2 so you may need to look at a few bits of historical commits to see the old vs new code.

 

A quote about isolated data stores from Eric Evans

Data Access

I need to keep this quote around so where better than my blog?

“For as long as I can remember, I’ve been advocating that a team that’s developing some complex piece of logic should have their own isolated data store and not have to share some huge database that has some kind of mishmash of different people’s ideas of what the data should be and so forth.”

(Eric Evans, “DDD & Microservices: At Last, Some Boundaries!” GOTO Berlin Dec 2015 (gotober.com) https://www.youtube.com/watch?v=yPvef9R3k-M)

This is such a difficult thing for so many developers to buy into. I frequently share a story about Eric trying to ease me out of a near mental meltdown, explaining to me when I was struggling with this (& I am paraphrasing and possibly adding some of my own comprehension at this point) that you have to pay the price somewhere and often its just less painful to solve the problem of having data in different places that needs to connect once in a while than it is to deal with the complexity of a system that tries to provide all answers to all of the problems. We have patterns to solve the “connect the data once in a while” problem (e.g. message queues).

I admit that I have to advise people to aim for this but it’s often not practical (or cost-effective) so if we’re talking about relational databases then we can at least isolate with schemas. But I do try to start with “separate database” and work our way backwards from there if it’s just not doable (or if the person or team I’m talking to looks like they are about to murder me). However each step backwards comes with a price. You just have to choose – pick your poison. As long as you are doing so with the right information at hand to make those decisions & choices, I think that’s the most important part.

ICYMI, My General Advice re DDD, EF & Domain Models with or without Data Models

Data Access, DDD

I wrote this in a Github issue thread about EFCore & lack of current support for complex types (which support DDD Value Objects .. to a degree) this morning and someone said I should share it so here it is:

I love value objects. I’m just not considering efcore for any serious work yet so I am not going to stress out about not using them. I find myself designing my domain with them out of habit then having to undo. And if you are set on efcore, then +++ to @jnm2 ‘s point. I map from domain to EF’s data model (the in-memory model that EF infers at runtime) using EF’s mapping when it’s easy and I don’t have to make annoying (or worse) concessions. Otherwise it is time for a mapper between domain and data model defined by separate classes + EF DBContext. So EF mapping layer is my default mapper when using EF. But sometimes it is just not sufficient and I go to (or recommend) the effort of building a separate data model. Your choice. Pick your battles.

Another point is that Jimmy Bogard doesn’t even think the ef6 complex types do enough to map types that you’ve defined as value objects. So he’s been either just using one to one or going the extra mapper route depending on the situation. Gogglebing Jimmy Bogard, something like “what’s missing in ef6 for DDD” to read more. And then there’s that collection conundrum, too as well as the problem of mapping private fields. So if you are okay without those in EF6, maybe support for value objects isn’t such a huge deal except for the problem of it being “taken away” temporarily.

My ref to Jimmy Bogard was for his Domain modeling with Entity Framework scorecard post.

I’m sharing because of the following reply from Joseph Musser on this thread:

I would love to have had (and still would love) an article that thoroughly addresses that question: weighing the pros and cons of staying within EF modeling constraints or using a data model and mapping yourself. When to consider making the switch. And what that means for creating a rich, disciplined model that doesn’t become anemic. I feel like I’ve wasted a lot of time not having a fundamental understanding of this from the beginning.

OSX, ASPNetCore, EFCore and CoreCLI, oh my!

Data Access, , , ,

After moving some RC1 test projects from windows over to my MacBook, it was time to start from scratch in OS X to see what that experience was like. Installing all the right pieces. I’d been using Visual Studio Code already on windows for nodes programming, but doing that for an ASP.NET 5 project is a little harder since the debugging for that isn’t implemented yet. At the same time I’m still getting used to navigating my way around a Mac … keystrokes, bash commands etc.

But I did get a small sample worked out and even used PostgreSQL to do the job.

That (remember, RC1) little test is on github here: github.com/julielerman/ef7osxtest.

But RC2 is a different beast!

Addendum because so many have asked: I am using the nightly builds of RC2. It is not out yet!

DNX is transitioning to CoreCLI with new underlying APIs. And there was that name change. ASPNET 5 became ASP.NET Core and EF7 became EF Core. The package and namespaces have changed.

And in the meantime, EF7/EFCore is still going through changes with RC2. To me the most significant is the work the team has been doing to help with disconnected graphs. You can read the latest (and I think final) state of how EF will handle disconnected graphs here on github.

I tried a few paths to starting a new project to try out RC2. Here’s some twitter evidence:

I had watched the video of David Fowler & Damien Edward’s talk about Core CLI from NDC London and David did the demos on a mac.

But it took a tweet from Tony Sneed to remind me that I could get David’s demos from github:

Indeed, that was the best starting point. I cloned that repository onto my MacBook and made sure I could run all three projects.

Now I’m working on building out the HelloMVC project by adding in the model, dbcontext, controllers and other relevant bits from my RC1.

At the moment (as of Feb 1 2016), the dotnet ef migrations commands aren’t working but Brice Lambson is working hard on it and says that should be pushed up this week. (Watch this github issue.)

And it might be a while before the Postgres provider gets updated to work with the new namespaces and package dependencies  but we do have the Microsoft.EntityFrameworkCore.Sqlite provider that will work on OS X although there seem to be a lot of problems at the moment with that on Linux. But I’ll be trying that out anyway.

EF7 is now EF Core 1.0, Package and namespaces change too

Data Access, ,

The big news last week was that the ASP.NET 5 & EF7 stacks were renamed in an attempt to alleviate the confusion around the naming of everything under the ASP.NET 5 umbrella.

ASP.NET 5 is now ASP.NET Core 1.0

EF7 is now EF Core 1.0

The naming has been a topic of debate since each of these were introduced by Microsoft and throughout their development. The teams finally accepted that the myriad questions and complicated answers about the naming in the ASP.NET 5 stack was enough of a red flag to change the names. While it is a huge relief to many, the timing is problem for people who have invested in asp.net 5 already because the name is not just a brand but it will affect packages as well. As per the announcement on github

  1. The Microsoft.AspNet.* packages and namespaces are changing to Microsoft.AspNetCore.*.
  2. The EntityFramework.* packages and namespaces are changing to Microsoft.EntityFrameworkCore.*.
  3. The version numbers of all of the above are being reset to 1.0.0-*.

Changing the namespaces this late in the game is going to be painful for devs/companies who have already been building apps and tooling on ASP.NET 5 &/or EF7. We are already on RC1 which was released in November with a go live license. Rick Strahl has a great [balanced] post talking about this.

Other than this, both ASP.NET Core & EF Core will move forward as planned. RC2 is supposed to be out sometime in February and they remain committed to the initial RTM at the end of March.

Here are the new namespace names for EF Core for RC2, as per https://github.com/aspnet/EntityFramework/tree/dev/src







In EF7
In EF Core




EntityFramework.Core
Microsoft.EntityFrameworkCore




EntityFramework.Commands
Microsoft.EntityFrameworkCore.Commands




EntityFramework.MicrosoftSqlServer
Microsoft.EntityFrameworkCore.SqlServer




EntityFramework.MicrosoftSqlServer.Design
Microsoft.EntityFrameworkCore.SqlServer.Design




EntityFramework.Relational
Microsoft.EntityFrameworkCore.Relational




EntityFramework.Relational.Design
Microsoft.EntityFrameworkCore.Relational.Design




EntityFramework.InMemory
Microsoft.EntityFrameworkCore.InMemory




EntityFramework.Sqlite
Microsoft.EntityFrameworkCore.Sqlite




EntityFramework.Sqlite.Design
Microsoft.EntityFrameworkCore.Sqlite.Design






If you are curious about what’s in EF Core, check out my EF7 Course on Pluralsight which I created during the Beta 4 …then read my recent post, EF7 Updates and Changes on Pluralsight’s blog which takes you from there to the current RC1.

	

	
	


							


	

		
				

			EF7 RC1 Notes–An Update to my Pluralsight “Planning Ahead for Entity Framework 7” course
		

		
		

			Data Access					

	


		

		
In early 2015, I published a course titled “Looking Ahead to Entity Framework 7”. This was created using an early version of the work that the EF team was doing on the total refresh of Entity Framework. 


That version was referred to as Beta 4 although was still so malleable that the EF team even referred to it as an alpha. In fact, the only reason it had the tag “beta 4” was to align with the set of Nuget packages that were being released for ASP.NET 5 as it was developing. An important goal with EF7 was that it needs to work with ASP.NET 5, so the EF team needed to release their stable pre-releases on Nuget.org (e.g. http://www.nuget.org/packages/EntityFramework.Core/) in sync with ASP.NET 5. The nightly builds, which are available on https://www.myget.org/gallery/aspnetvnext are a different story and are pushed frequently by the EF team whenever they are ready to do so. 


Since the course was released some technical things have changed and even the release plans have changed. Rather than updating the course each time, potentially introducing more things that will change by RTM, I will wait until EF7 is closer to RTM and use that to create a course about the release. 


In the meantime, I think it is important to have some awareness of the current landscape so that combined with watching Looking Ahead to Entity Framework 7, you can plan ahead effectively. My recommendation is to at least peruse this blog post before watching the course and then come back and read it more carefully when it will make more sense. 


In this post I will address each of the 6 modules of the beta 4 course and alert you to what has changed and what to expect. 


Module 1: Achieving Microsoft’s Goals for EF with EF7


The overarching goals for EF7 have not changed. Microsoft is wholly committed to a completely new code base for EF7 using modern software practices. The New Devices New Data Stores goal is still correct and as you learned (or will learn when you watch the course), the proof of concept work around NoSQL and Redis is still set aside. 


While discussing where you can use EF7, I showed a slide that talked about various frameworks. 


clip_image001 


This shifts a bit since the initial RTM of EF7 will now support UWP (Universal Windows Platform) which targets Windows 10 devices including phones and tablets. Along with this change, the SQLite provider that had been set aside, is now part of the initial RTM. In the course, I explained that the Windows Phone 8 & Windows Store support is also set aside for a later release after the initial RTM. That is still true.  


Here is an updated version of that slide I created for a recent conference: 


clip_image002 


Another change to the first module is related to the timeline. I explained how EF7 was going to focus on being able to release alongside of ASP.NET5 which meant setting aside of of the goals that they wanted to achieve. I also explained that when ASP.NET 5 was released, we would get a release of EF7 that would be still called a pre-release, not an RTM. Here is the slide I used while discussing this timeline. 


clip_image003 


These plans have changed somewhat. While the initial ASP.NET focused release of EF7 is still going to be a subset of all that the team wants to implement in EF7, it is now going to be a full release, an RTM, not a pre-release. So at the same time ASP.NET 5 is released, EF 7.0.0 will be released.  


Here is what has not changed however: 


· EF 7.0.0 will still be an “abbreviated” version.  


· The features that the team is focusing on are those which align well with web applications. A good example of this is that they are working on improving the experience with disconnected entities and have set aside some magical relationship features such as explicit and lazy loading.  


· EF 7.0.0 will not become the “official” version of Entity Framework.  


· EF6 will continue to be the official version.  


· EF6 will continue to be what’s delivered when you install entityframework via Nuget.  


· Your apps that use EF6 will not automatically update to EF7!  


· When installing the packages via Nuget, you will not need to use “-pre” to distinguish between EF6 and EF7. 
EF6 will continue to install using install-package entityframework. There is no plain old entityframework package for EF7. You would begin with your desired provider (e.g. entityframework.inmemory) and that will pull in all dependences e.g. entityframework.core and more. Since this is a full RTM for EF7, you will no longer need to use the –pre. 


Also, we now have a pretty well-delineated roadmap from the EF team on their github wiki. Most importantly is that the RC1 (Release Candidate with Go-Live) version of EF7 (as well as ASP.NET5) was released in November 2015. According to the ASP.NET 5 roadmap, there will be an RC2 in February with the RTM targeted to the first quarter of 2016. There will be a big change Another notable point on the ASP.NET 5 website is the lack of Visual Basic support until possible Q3 of 2016. 


According to the roadmap, EF7 will become the official version at some point in the future when the team feels that they have a critical mass of important ORM features implemented. 


Here is an updated version of the above timeline slide which I used in my recent conference talk: 


clip_image004 


Module 2: Targeting EF7 Initial and Future Releases


The list of things collected in what will be in the initial release has not changed significantly except that now, as mentioned above, EF7 will also support the UWP apps. 


Re VS *& .NET versions. While ASP.NET 5 can run on the DNX environment, it can also be run on a full .NET Framework environment. In the course, it says this will be possible with 4.5 and beyond. Now it will be for .NET 4.5.1 and beyond. I don’t believe that the future releases will revert back to supporting 4.5 at all. 


Installing: As mentioned above, the latest stable release is available on Nuget.org. (Today that is RC1). The nightly builds are available via the myget package source (https://www.myget.org/gallery/aspnetvnext). The team is currently working on RC1 in the nightly builds. Sometime this month (November 2015) a stable version of RC1 will become available on Nuget.org and that will be what you get with the “-pre” tag. 


In Beta4 there was still an EF7 package named EntityFramework. That no longer exists, enabling us to use that package name to specifically target EF6. For EF7, it’s easiest to begin with the provider that you want and then this will pull down all of the relevant dependencies. So for example 
install-package entityframework.sqlite” will pull down that package, the relational package (which includes migration support), the core package and others.  


Don’t forget that the migrations commands are in a separate package: EntityFramework.Commands, which you need to explicitly install if you want to perform migrations in nuget or the DNX environment. 


This is important: the “K Runtime” is now the “DNX runtime”. 


All of the commands that you run start with dnx now, not k. 


For example:


dnx ef migrations add



Playing with the proof of concept features The support for the stripped down framework used for Windows Phone 8 and Window Store apps was set aside and is still in that state. Those DLLs are not handily available. The tricks I used in the course to access those assemblies may or may not work still. I haven’t tested. Remember though that the SQLite assembly is now part of EF7 so that’s easy to get! And if you can target Windows 10 phone & tablet apps with EF7.


Module 3: Querying and Updating with EF7 (Disconnected Graphs change!)


Most of the changes from that evolved since I created this course with beta4 are around syntax. Even in the course you could see that at first the team was renaming methods to better suit how they wish they had been named. For example, rather than have DbSet.Add and DbSet.AddRange, they modified add to just take an overload that accepted a range. But just before I pushed the course live, they changed the methods back to Add taking a single object and AddRange specifically for a range. The reason for this is to lessen the blow of changes to the API. This could be something that would be backwards compatible with EF6. There have been a number of changes like this as EF7 has evolved where some of the syntax was reverted to better align with EF6. Even behavior has shifted. Add is another example. EF6 and earlier had a pattern of [almost] always affecting full graphs when you pushed a root of a graph into the context with Add/attach/Delete or using the Entry().State property. And the effect were inconsistent. They experimented with completely separating behavior by making Add (etc) and Entry.State ONLY affect the root and then giving us a new method ChangeTracker.AddGraph soley for working with graphs. Since beta4, the team has narrowed in on a pattern that will be the final say on this matter. Here it is:


DbSet.Add, AddRange, Attach, AttachRange, Update, UpdateRange: These methods now take a new 2nd parameter which is an enum,GraphBehavior. 


GraphBehavior values are IncludeDependents and SingleObject.


The default for the above methods is GraphBehavior.IncludeDependents which means that a just using it in a familiar way, e.g.:


context.Samurais.Add(myNewSamurai);



will result in *almost* familiar behavior. THe catch (which is a good improvement, in my opinion)is that the default behavior, IncludeDependents, is literally for dependent objects. it is not going to include All Related Objects. The distinction is important. Objects in a relationship are either principal (aka “parent”, or dependent (aka “child”). In the database these are easily distinguishable because the dependent is the one that has the foreign key back to the principal.


Consider the following Principal to Dependent relationships:


Order –> Line Items


Person->Address


Category->Products


It makes sense to create a new order, add some new line items to it and then add that order graph by calling context.Orders.Add(order). In this case, all of the dependent line items will be included in the operation, i.e. marked as Added as well.


It makes sense to add an address record to a person and then add that person to the context. context.People.Add(person). Again, the dependent address  would be marked as added. The category with it’s dependent products also is logical.


If for some reason you created a new line Address instance then identified it’s Person by setting someAddress.Person=somePerson, and then added that address via context.Addresses.Add(someAddress), EF7 will not mark the parent/principal object (somePerson) as Added. Adding a new address does not necessarily mean you are adding the new person so EF won’t make that presumption. I also am a fan of building my model with aggregate roots that are in charge of the behavior of their related data. So I wouldn’t allow my tyeps to be used in such a way that the user of my API could create an address and randomly add it anyway. So this behavior aligns with coding patterns that I recommend.


It also solves another problem I see a lot where devs use object instances to specify reference properties. I always recommend using foreign key properties for this but I know that so many programmers have the following problem.


Imagine this scenario:


The form for building or editing the order has a place to enter the a new  shipping address for a customer. One of those fields is a dropdown for “region” (in my case, in the U.S. that would be a state like Vermont or Utah). I query for a list of region objects and populate a dropdown list. 


clip_image005


I select Vermont and then my code does something like this: theOrder.theAddress.Region=(region)List.SelectedItem. I add the order with context.Orders.Add(theOrder) and call SaveChanges.


The next person who goes to add an address sees this:


clip_image006


Two Vermonts! That’s because in EF6 (and earlier) everything in the graph is marked added and the EF inserts the Vermont object into the database even though it was already there.


In this case, the model sees REGION as the principal and ADDRESS as the dependent because ADDRESS has a foreign key back to REGION. EF7 will not include the principal. Only the address will get inserted and the crops are saved!


Add, Attach & Update: Root Only


So, the default for Add, AddRange, Update, UpdateRange, Attach and AttachRange is to include dependents. Using the parameter, you can specify the SingleObject enum and then only the root entity of the graph will be affected by the method.


DbContext.Entry().State


In RC1 this is the same as I explained in the course: if you specify an entity that is in a graph, only that entity will be affected by setting the state. Any other objects that are part of the graph will be untouched.


ChangeTracker.AddGraph has changed to ChangeTracker.TrackGraph


The signature is the same, you pass in the graph and a lambda function. The lambda can express the state that you want the graph to use.


context.ChangeTracker.TrackGraph(someEntityWithRelatedObjectsAttached,          e => e.Entry.State = EntityState.Added);


EF will walk the graph and apply that function to everything it discovers in the graph, skipping objects that are already being tracked (and their related objects).    A cool feature of this method (same as when it was called AddGraph) is that that function does not have to express state. It can be any function you want.


The DbSet.Find method
This was set aside for post-RTM, but the team is reconsidering and may get it in for RTM. More here: https://github.com/aspnet/EntityFramework/issues/797


Module 4: Using and Migrating Relational Databases


In this module I talked about some differences between how migrations worked relative to how we are used to them working in EF 4.3 to EF6. I also showed how migrations work in the k runtime commands if you are using ASP.NET 5 and not able to use the familiar PowerShell commands. I also explained that the magic behavior of the DbInitializers as well as automatic migrations will not be part of EF7.


Not a lot has changed since the beta4 release that the course is based on. The team has streamlined the commands a bit more. They had introduced a new “apply-migration” command to replace update-database, but apply-migration is gone and you will just use the familiar update-database command.


On the ASP.NET side, the commands are now dnx commands, not k commands. So the current way to express adding a migration, therefore is:


dnx ef migrations add myAwesomeMigration



Not much else has changed from what is explained in Module 4.


There’s a nice chart in the July 23rd EF design meeting notes that shows the changes.


Not really migrating, but the commands now support reverse engineering with a scaffold command that has a bunch of helpful parameters to customize how the code is generated.


Module 5: EF7 Futures


This module covers things that the team was exploring but set aside to focus on the ASP.NET 5 aligned release. Those were non-relational stores (with Redis and Azure Table Storage as their proof of concept) and allowing EF to run on the trimmed down version of .NET that is used for Windows 8 phone and tablet (aka “windows store”) apps which also relied on the new SQLite provider. Since the assemblies were still accessible, I showed a demo of a Windows Phone and Windows Store app that used SQLite as well as one that used Azure Table Storage.


The Azure Table Storage and Redis providers are still set aside, as I explained in this module.


And while it is still true that EF7 will not initially support this Windows 8 Phone and Store apps when it is released, the team did bring SQLite back into the fold for EF 7.0.0. The reason that SQLite was re-ignited is because, as I mentioned above, EF7 is now able to run on UWP (Universal Windows Platform) the Windows 10 platform that lets us create cross-device apps that will also run on phones and tablets. Those will benefit from using SQLite. In fact, I was able to rebuild the Cookie App from this module in UWP with EF7 and SQLite and watch it run on emulators for phones and tablets.


Module 6: Interacting with the Team


This module is about how EF7 is being developed openly on GitHub at github.com/aspnet/entityframework and the story has not changed. The team continues to be eager to have developers try out EF7 and provide feedback in the form of issues or even Pull Requests. They continue to publish their team meeting notes on the wiki where we can converse with them further about their ideas.


The EF7 RoadMap


The roadmap did not exist when I created the course. It is a great resource and I encourage you to check it out at https://github.com/aspnet/EntityFramework/wiki/Roadmap.


I continue to bang on EF7 and pester the team when I’m confused. When EF 7.0.0 releases, I will create a Getting Started course that will dig further into EF7 as a full released framework.


A Few Notes About Mappings


Since the course there were a few things I learned that either didn’t exist at the time or I just didn’t know about that I think are worth mentioning.


The pluralization support we’ve been used to in the past does not exist in EF7. If you have a model with an entity named Person and another name Order, EF (& migrations) will presume that the relevant tables are also named Person and Order. 


The next point will help you address the pluralization.


Custom Conventions that we got in EF6 are not in RC1 and won’t be in the first itereation of EF 7. They are targeted towards a future release — and should be in EF7 by the time it becomes the “official” version of EF. 


So a custom convention such as this is not yet possible:


(EF6->) modelBuilder.Properties<String>().Configure(p => p.HasMaxLength(50))


However, Rowan Miller points out in his gist (https://gist.github.com/rowanmiller/88261afd0baae7fb9b04) that we can continue to use a hack that we had before custom conventions existed to help (but not magically solve) with table name pluralization. And thanks to the new Name property introduced in C#6 (& VB but remember, we don’t have VB support yet) you could iterate through the entities and apply someo rules. This gist of Rowan’s shows a simplistic “add s” rule:


foreach (var entity in modelBuilder.Model.GetEntityTypes())


{


modelBuilder.Entity(entity.Name).ToTable(entity.Name + “s”);


} 


Support for Table per Hierarchy (TPH) mappings was added as of RC1.


EF7 can now infer 1:1 mappings without you having to specify the principal and dependent. If it doesn’t get it right then you can use updated HasMany/HasOne fluent mappings. (These have been simplified!)