All posts by Julie Lerman

Quick Tips for Migrating EF Core at Runtime on Azure

Data Access

A question on my EF Core 2 Getting Started course on Pluralsight asks:

Hi Julie
Thank you for your course. I have a question. Can you please advise what is the best practice to deploy code first migration to the production database (azure).? I mean i have created asp.net core mvc and ef.core application using the code first migration approach. after deployed to azure. If i change any schema in our code first approach how can i update schema in production database ( without loosing the data in production database)

My response with a few ideas I thought were worth sharing outside of the course discussion. (Note that this is just high level)

For simple solutions, one path I have used is to call database.migrate at startup. The VS publish workflow has an option you can check to apply migrations when the app is published. You can see this in the MS docs for deploying an aspnet core app to Azure:

Or you can do it programmaticly in the program.cs file which will perform any needed migrations. There’s an example of this in my April 2019 MSDN Magazine article . If you need something more robust, then you could instead generate scripts (perhaps Idempotent scripts) with EF Core migrations and then include them with your updates and use a tool that can apply those scripts. If you use Redgate tools, perhaps their SQL Change Automation tool. Another type of tool is a migrator tool like FlywayDB (flywaydb.org) or LiquidBase. I’ve used Flyway with containers. Here’s a very recent conference talk I did where I used it: bit.ly/30AhgAr

Publishing a Single Image Docker Container with Secrets from VS2017 and Running it on Azure

Docker

(Written in advance, but published on May 2 when the relevant article is finally available.)

I’ve just finished writing a three part series on building a containerized ASP.NET Core API that uses EF Core for its data persistence. All of this was done in VS 2017 and I took advantage of the VS2017 Tools for Docker.

The article series will be in the April, May and June issues of MSDN Magazine.

Part 1: EF Core in a Docker Containerized App, Apr 2019

Part 2: EF Core in a Docker Containerized App, May 2019

But I didn’t have room to include the important task of deploying the app I’d written, although I worked hard to do it. Well, the deployment was pretty easy but there were some new steps to earn in order to deal with storing a password for making a connection to my Azure SQL database. I will relay those steps in this blog post.

My API uses EF Core and targets an Azure SQL database. So whether I’m debugging locally with IIS or Kestrel, debugging locally inside of a Docker container or running the app from a server or the cloud, I can always access that database.

That means I have a connection string to deal with but I want to keep the password a secret.

The structure of the solution is here. My ASP.NET Core API project is DataAPIDocker. And because I used the docker tools to add container orchestration, I have another folder in the solution for docker-compose.

image

I go into detail in part 2 of the article (the one in the May 2019 issue) but the bottom line is that I use a docker environment variable in my docker-compose.yml file.

version: ‘3.4’

services:
dataapidocker:
image: ${DOCKER_REGISTRY-}dataapidocker
build:
context: .
dockerfile: DataAPIDocker/Dockerfile
environment:
– DB_PW

In the environment mapping, I have a sequence item where I’m defining the DB_PW key but I’m not including a value. Becasue there’s no value there, Docker will look in the host’s environment variables. Because I’m only debugging, I create a temporary environment variable on my system with the value of the password and when I debug or run the app from VS2017, the password variable will be found. That environment variable gets passed into the running container and my app has code to read it and include that password in teh connection string.

So its all self-contained, nice and neat.

Publishing the Image to Azure’s ACI Registry

Once you’ve got the app working it’s time to publish it. But we’re using Docker, so you’re not publishing the app, but the docker image that can run the app for you. Docker tools will help with this also.

Right click the project and choose Publish.

image

Then you will want to create a publish profile. And part of that profile is to choose where to publish the image. Here you’ve got options. I have a VIsual Studio Subscription and can publish it to an Azure Container Registry if I want or to Docker Hub or to some other registry.

image

My goal for this blog post is to get the image into the Azure Container Registry so that’s my choice. You can have multiple container registries in your azure account. And you can store any number of images in a single registry. Well, there may be technical or financial constraints, but the point is that you can have multiple images in a registry. I’m not here to advise on how to manage azure finances, just how to do the task.

Here’s the overview page of a registry I let the publishing tool create for me.  I’ve circled the link to see repositories which is where your images are accessible.

image

You may have different versions of a particular image so each “set” is a different repository. I have three repositories in mine where I’ve been experimenting.

image

The dataapi has only one image which the publishing tool automatically tagged “latest” for me. I can have other versions under different tags.

Back in Visual Studio, after walking through the publishing tool’s questions for creating a new repository, the final step is to go ahead an publish which will build the image and push it up to the target repository. Keep in mind that you’ll want VS2017 to be set to run a RELEASE build, not a DEBUG build.

If its your first time pushing this image to the repository, the tooling will also push the ASP.NET Core SDK and runtime images that are listed in the app’s Dockerfile .

I was surprised to see this, wondering why Azure didn’t just grab them from docker hub and why I was uploading those big files directly. Naturally I tweeted my confusion:

There’s more to the story but it is beyond the scope of my goals here.

A cool feature of this registry is that you can right click and run an image. Which is fine if you aren’t trying to orchestrate a number of images and that matches my case. This image does run independently.

Right click on the image and choose Run instance. Azure will create a container and run it as an Azure Container Instance. Although first you need to define specs for the instance.

image

It’s kind of magical because you don’t have to create and manage a virtual machine to run the container on if it’s a simple application.

What About Environment Variables for the Container Instance?

The instance will run but the Magazines controller that needs to read from the Azure SQL database will fail because we haven’t provided the password which the container expects to be provided through the host’s environment variable. So for my image, right click and run wasn’t quite enough.

This is where I had to do a lot of reading, research and experimentation until I got the solution working. (Keep in mind that if I were running this on a virtual machine of my own devising, you can just pass the variable in when you manually call docker run.)

There are two ways to provide an environment variable to a container instance.

One, through the portal, means rather than right clicking the image, you need to start by creating a new container instance in Azure and pointing to the image. This path lets you assign up to 3 environment variables in the configuration:
image

Using an On-The-Fly Variable to Pass into the Container

I’m going to do a first pass creating an variable on the fly to pass to EnvironmentVariable. Then I’ll show you how to use the Azure Key Vault

EnvironmentVariable expects a hashtable.

Create a new variable (I’ll call it envVars in homage to the resource where I learned this) and assign a single key value pair:

$envVars = @{‘DB_PW’=’eiluj’}

The other tricky part is providing the credentials to access the image in the registry. We don’t have to do that when using the portal to create the container because we’ve already provided them. But now I need to provide them.

You’ll need the user name and password from the repository:

Then you can use PowerShell to create a secure string from the password and then use that secure string along with the username to create a PowerShell credential object.

TIP: If you have multiple subscriptions, be sure you’re pointing to the correct one where the target resource group is.

TIP: A cool thing you can do in cloud shell is type DIR to list your subscriptions and then use CD to get into the correct one! Checkout the PowerShell Cloud Shell quick start for details

.

TIP: If like me, you mess around with the database to experience cause & effect, remember that in my sample code, the database gets migrated on app startup. In the case of having it in a container that means when the container instance is run. So if you run the container, then delete the database, you won’t see the db again until the container is spun up again. Stopping & restarting has the same effect. Of course this is just for testing things out, not production! Once again, something that had me stuck for over an hour until I had my aha moment.

Creating a KeyVault and Adding My Secret Password

Links to My Recent DDD+EFCore Content

DDD,

Time to aggregate the various articles and videos I’ve created filled with lessons on how EF Core support for direct mapping of domain models to your relational databases has been improving:


A Small Lesson on env Files in docker-compose

Docker

I’ve been working a lot with docker lately and learning learning learning. I have written a 3-part series for my MSDN Mag Data Points column that will be out in April, May and June 2019 issues. I have another YUGE blog post that I will publish to accompany the May article. And I’m working on others.

I explored Docker environment variables and different ways to feed them into a Docker image or container.

My docker-compose file referenced an environment variable named DB_PW without specifying it’s value.

dataapidocker:
   image: ${DOCKER_REGISTRY-}dataapidocker
     build:
       context: .
       dockerfile: DataAPIDocker/Dockerfile
     environment:
       - DB_PW

Docker will read environment variables from the host to discover the value.

But this was a pain. I wasn’t permanently storing the DB_PW on my development machine and had to remember to set it frequently. Elton Stoneman (from Docker) said I should *really* consider using the Docker env file feature. This lets you set variables in an environment file and let the docker-compose file read from that. And I can keep that special file out of my source control. (Always my worry!)

I started by following docs that showed using a file named anything.env. I created a file called hush-hush.env where I specified the variable. This is the full content of the file:

DB_PW=thebigsecret

Then in docker-compose, in the service, the env_file tag lets you point to it. You can even remove the environment tag in the yml file.

dataapidocker:
   image: ${DOCKER_REGISTRY-}dataapidocker
     build:
       context: .
       dockerfile: DataAPIDocker/Dockerfile
     env_file:
       - hush-hush.env

This worked perfectly. My app was able to discover the environment variable in code.

But then I evolved my solution to use another container for my database. The primary container depends on the new mssql container. And the mssql container requires I pass in the database password as an environment variable. Since DB_PW already exists, I can do that easily enough with substitution (via curly braces). Here’s the new docker-compose file:

version: '3.4'
 services:
   dataapidocker:
     image: ${DOCKER_REGISTRY-}dataapidocker
   build:
       context: .
       dockerfile: DataAPIDocker/Dockerfile
   env_file:
     - hush-hush.env  
   depends_on:
     - db
   db:
     image: mcr.microsoft.com/mssql/server
     volumes:
       - mssql-server-julie-data:/var/opt/mssql/data
     environment:
       SA_PASSWORD: "${DB_PW}"
       ACCEPT_EULA: "Y"
     ports:
       - "1433:1433"
 volumes:
   mssql-server-julie-data: {}

And there’s an order of operations issue here. When I build the docker-compose file, it complains that DB_PW is not available and my app fails. The db service is not getting the contents of my hush-hush.env file. I tried a number of things, such as adding env_file to the db service. In the end, here’s what I learned.

The substitution use requires that the DB_PW variable be defined in docker-compose. I added that back in to the primary service, but it was not getting the value from hush-hush.env.

But you can have a .env file that has no name. The extension *is* the full name of the file. Docker-compose will read that early enough and provide the value from the .env file to the declared DB_PW. Then all of the pieces fell in place. The mssql container was spun up with the value from DB_PW as its environment variable. And my app code was able to read the environment variable that Docker passed into the running container for its own tasks.

The final docker-compose.yml file looks like this:

version: '3.4'
 services:
   dataapidocker:
     image: ${DOCKER_REGISTRY-}dataapidocker
   build:
       context: .
       dockerfile: DataAPIDocker/Dockerfile
   environment :
     - DB_PW 
   depends_on:
     - db
   db:
     image: mcr.microsoft.com/mssql/server
     volumes:
       - mssql-server-julie-data:/var/opt/mssql/data
     environment:
       SA_PASSWORD: "${DB_PW}"
       ACCEPT_EULA: "Y"
     ports:
       - "1433:1433"
 volumes:
   mssql-server-julie-data: {}

And it relies on a file named “.env” with my variable key value pair defined (same as hush-hush.env above).

DB_PW=thebigsecret

Resources:
https://docs.docker.com/compose/environment-variables/

Announcing: Deep Dive into EF Core 2-Day Workshop

Conferences, ,

Join me in London June 17-18 for a 2-day deep dive into Entity Framework Core.

This is a new addition to Skills Matter course catalog. Because it is a new course, we are looking to get feedback on the proposed list of topics to be covered. If you are interested in attending, your input will be helpful.

Is the list of topics too long for 2 days? Does it touch on what you would want to learn in an advanced class? You can provide feedback on the course description page.

Day 1: Leverage Advanced Features

  • High level review of EF Core differences from EF6
  • Implementing logging to capture EF Core’s database and in-memory activity. Learn about different types of logging data to be captured
  • Learn various approaches to seeding such as via database scripts, code or using the migration-based seeding introduced in EF Core 2.1. You’ll also learn when each approach may be appropriate
  • Using migrations during development,within source control and during deployments
  • Integration testing your EF Core code

Day 2: EF Core in Your Software Architecture

  • The Great Repository Debate: Pros and Cons of the Repository Pattern/Generic Repositories for exposing EF Core
  • Designing Data Layers/APIs
  • Understanding complicated mapping conventions and supplementing those with custom mappings using the Fluent API
  • Designing for performance
  • *Bonus topic* If all modules have been covered, you will also look at EF Core in Azure Functions and EF Core with Azure Cosmos DB (given adequate time)

A Few Coding Patterns with the MongoDB C# API

Data Access

In the February 2019 issue of MSDN Magazine (Exploring the Multi-Model Capability of Azure Cosmos DB Using Its API for MongoDB), my Data Points column explored working with the MongoDB model of Azure Cosmos DB using the mongocsharpdriver. I started by working against a local instance of MongoDB and then the Azure instance. But the column was a little bit long so I cut out a few extraneous sections . So I’m placing them here and linking to this blog post from the article.

In the article I used an IMongoCollection object to query and store data into the database. You must specify a type for the collection object to serialize and deserialize. In the article I typed the collection to my classes, e.g.,  Collection<Ship>. It’s also possible to type the collection generically to a BsonDocument. Here’s some information about that and a little bit of code.

Typing Collections to BsonDocument

Another path for mapping is to use a BsonDocument typed collection object that isn’t dependent on a particular type. This would allow you to have more generic methods. But it also means manually serializing and deserializing your objects, which is easy using ToBsonDocument for serializing:

var coll = db.GetCollection<BsonDocument> ("Ships");
coll.InsertOne (ship.ToBsonDocument());

Given that the documents have discriminators, you can then specify a type in your query to retrieve specific types although, by default, hierarchies don’t get accounted for. The article refers to documentation on polymorphism for the C# API. Here’s the link. Check  to learn how to properly implement polymorphism in more detail . The following code will only pull back documents where _t matches the configured discriminator for Ship into ships and for DecommissionedShip into dShips:

var coll = db.GetCollection<BsonDocument> ("Ships");
var ships = coll.AsQueryable().OfType<Ship>().ToList();
var dShips = coll.AsQueryable()
                  .OfType<DecommissionedShip>().ToList();

Encapsulating the MongoClient, Database and Collection

Specifying a typed collection instance repeatedly, as I did in the article demos, can become a drag. You could set them up in advance, for example in a class that acts as a context for interacting with the database, as shown here:

public class ExpanseContext
{
  public IMongoDatabase ExpanseDb { get; private set; }
  public IMongoCollection<Ship> Ships { get; private set; }
  public IMongoCollection<Character> Characters {get;private set;}
  public ExpanseContext()
  {
    ExpanseDb=new MongoClient().GetDatabase("ExpanseDatabase");
    Ships=ExpanseDb.GetCollection<Ship>("ships");
    Characters=ExpanseDb.GetCollection<Character>("ships"); 
  } 
}

This refactored code to insert a document is much more readable:

private static void InsertViaContext ()
{
  var context = new ExpanseContext ();
  var ship = new Ship { Name = "Agatha King" };
  context.Ships.InsertOne (ship);
}

Logging in EF Core 2.2 Has a Simpler Syntax–More like ASP.NET Core

Data Access,

Logging EF Core’s memory operations and SQL operations has evolved a few times since EF Core arrived. It takes advantage of the same underlying features that ASP.NET Core uses. If you are using ASP.NET Core, logging is baked in and it is really simple to turn it on for EF Core and add filtering. See Shawn Wildermuth’s blog post about EF Core logging in ASP.NET Core.

But if you aren’t using ASP.NET Core, it’s a little more complicated. Not terribly, but still there’s some extra work to do. It involves setting up an ILoggerFactory in your DbContext and defining any filters at the same time.

I wrote an article about this (with the focus being on taking advantage of the various available filters for EF Core logging) in MSDN Magazine earlier this …oh wait, it’s Jan 1, so I can say “last  year”.  Data Points – Logging SQL and Change-Tracking Events in EF Core. I also used it heavily in my EF Core 2 Getting Started course, EF Core 2:Mappings and EF Core 2.1: What’s New courses on Pluralsight. (Note that I’ve updated the sample code for the Getting Started course to EF Core 2.2 and put it on GitHub at github.com/julielerman/PluralsightEFCore2GettingStarted)

My article and courses were using Console apps to demonstrate EF Core behavior and therefore the ConsoleLoggerProvider to tie the logger to the console. Note that the Data Points article contains a lot of good details about the various types of filtering. So you can use the new syntax (below) to specify that there should be a filter, but be sure to read the article to learn about the flavors of filtering and what type of details you’ll be able to see based on the choices you make.

But the logging API has continued to evolve and is providing some of the same shortcuts that ASP.NET had created. And the ConsoleLoggerProvider has been deprecated. The API is not part of EF Core. It’s part of .NET Core. Both EF Core and ASP.NET Core use it.

If you are using EF Core 2.2, the syntax has changed (simplified) and it’s going to get even more streamlined in 3.0.

In fact, if you use the earlier syntax with 2.2, you’ll get a warning about the ConsoleLoggerProvider:

Obsolete(“This method is obsolete and will be removed in a future version. The recommended alternative is using LoggerFactory to configure filtering and ConsoleLoggerOptions to configure logging options.”)

For a point of comparison, here is an example of using theold syntax to turn on logging, only show logs related to database commands and only show messages that are tagged as “Information”.

EF Core 2.0 & 2.1 Logic

public static readonly LoggerFactory MyConsoleLoggerFactory
            = new LoggerFactory(new[] {
              new ConsoleLoggerProvider((category, level)
                => category == DbLoggerCategory.Database.Command.Name
               && level == LogLevel.Information, true) });

Once your logger factory field is defined in the context class you tell the DbContext to use it when configuring.

protected override void OnConfiguring
  (DbContextOptionsBuilder optionsBuilder)
{
  var connectionString = 
    ConfigurationManager.ConnectionStrings["WPFDatabase"].ToString();
  optionsBuilder
    .UseLoggerFactory(MyConsoleLoggerFactory)
    .EnableSensitiveDataLogging(true)
    .UseSqlServer(connectionString);
}

So it’s the creation of the logger factory whose syntax is a little convoluded. The newer API follows how ASP.NET Core lets you filter with an AddFilter method that takes the filters as parameters. No lambdas needed. Also configuring the filter is a separate bit of logic that tellig the logger that it should be tied to the console.

EF Core 2.2 Logic

With EF Core 2.2, you can set up the logger factory in the constructor or another method as long as it’s available when you are configuring the option builder. I’m creating it in a method then using that method as a parameter of UseLoggerFactory. I’m still filtering on showing only database commands and log details flagged as Information.

private ILoggerFactory GetLoggerFactory()
{
  IServiceCollection serviceCollection = new ServiceCollection();
  serviceCollection.AddLogging(builder =>
         builder.AddConsole()
                .AddFilter(DbLoggerCategory.Database.Command.Name, 
                           LogLevel.Information)); 
  return serviceCollection.BuildServiceProvider()
          .GetService<ILoggerFactory>();
}

and then I’m calling GetLoggerFactory() in the UseLogging method on the optionsbuilder:

optionsBuilder.UseLoggerFactory(GetLoggerFactory())

Packages and References

In order to use the AddConsole() method, you still have to use the Microsoft.Extensions.Logging.Console package that the earlier ConsoleLoggerProvider was in. However, you do not need a using statement for the namespace (as you did for the ConsoleLoggerProvider).