Tag Archives: docker

Exploring docker compose watch in Visual Studio Code

Docker’s compose watch feature was first released in Sept 2023 as part of docker compose v 2.22.0 (https://docs.docker.com/compose/release-notes/#2220)

Here are some tips and behaviors to be aware of when using compose watch in Visual Studio Code as well as how they relate to compose up.

While it shouldn’t make a difference, note that I’m using MacOS Monterey, latest VS Code (1.90.2) and latest (official Microsoft) Docker extension for VS Code (v1.29.1).

My app is a tiny dotnet web app using a minimal API. I’ve also performed the same experiments using a more complex app (with a database and FlywayDB containers and using controllers instead of minimal API). In the minimal API,  one of the mapgets is defined in the program.cs file, the other is in a endpoints file and connected using a method to add those to the API. I did this to see if that made a difference in recompiling. More on that further on.

With a combination of “instructions” in your docker-compose file and the watch command, Docker will keep an eye out for code changes in the relevant container(s) that you have specified for watching , then update and rebuild the container on-the-fly. It’s an incredible improvement over having to explicitly take down the container (whether you stop or remove) then spin it up again to try out the changes. And thanks to the efficient caching in docker compose (the v2 compose, not the older docker-compose command) it can do so very quickly. Sounds magical, right?

Container updates? What?

How do I know the container with my app got updated? Two ways. The docker terminal in VS Code outputs all of the rebuild details and  I leave the web page open after calling my API Get the first time and refresh the page after the container is rebuilt to see the new response data.

The Repo

The solution I’m using for testing can be found at https://github.com/julielerman/DockerComposeWatchDemoMinimalAPI.

Setting up docker-compose for watching

The key in the docker-compose file is a service subsection called watch inside the develop section for whichever service (container) you want to be watched. Below is a simple example.

I want watch to update the valuesapi container if I change any of the API code. I’ve added a minimal develop section in my docker-compose.yml file to support that in the API service. You can learn more about the various attributes in the develop specification doc (https://docs.docker.com/compose/compose-file/develop).

    develop:
     watch:
        - action: rebuild
          path: .

Here is the full compose file so you can see where this section sits within the service.

services:
  valuesapi:
    image: ${DOCKER_REGISTRY-}
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 5114:5114
    develop:
      watch:
        - action: rebuild
          path: .

Lesson Number One: Control Autosave Settings

I have always had autosave enabled with the “afterDelay” option in Visual Studio Code. And I’ve just allowed the default timing afterDelay at 1000 milliseconds. I quickly learned that AutoSave was a big problem as I was working on some edits and Docker kept updating the container over and over again as I was typing and thinking. Not a favorable situation.

The other options are

  • onFocusChange (when the editor loses focus)
  • onWindowChange (when the window loses focus)
  • off

I initially just completely disabled AutoSave with off, but that is also dangerous since I’m so used to relying on it. Instead I changed the setting to onFocusChange which seemed most reasonable to me. I still find myself needing to explicitly save but expect I’ll get used to it.

How Quickly is the Container Updated?

Before I discuss the docker compose syntax, I do want to give you a sense of how quickly Docker updates a container.

I’m only using the docker compose watch command for the tests in this section.

Obviously, everyone’s performance will be different with respect to their setup so YMMV. But the results should be relative.

To begin with, calling docker compose watch takes no more time than docker compose up.

Keep in mind that a change to my dotnet web app means that dotnet needs to rebuild the app. If it’s just code, that’s simple.

I noticed that the first change I made that triggered Docker to recreate the build and publish image layers.

=> [valuesaapi build 7/7] RUN dotnet build "ValuesTest.csproj" 11.0s
=> [valuesaapi publish 1/1] RUN dotnet publish "ValuesTest.cspro 3.4s

The entire rebuild was 15.2 seconds before the API reflected the change.

Yet on subsequent changes, the rebuild was only 0.5 seconds and those two layers came from the cache.

=> CACHED [valuesaapi build 7/7] RUN dotnet build "ValuesTest.cs 0.0s
=> CACHED [valuesaapi publish 1/1] RUN dotnet publish "ValuesTes 0.0s

I have no idea how this is working (how the change is getting to the container layer) because the change was, indeed, reflected in the output from the API.   I can’t explain this difference but will be asking about it for sure! (Then will come back to edit this post.)

Here is a look at the docker log for the rebuild of the version of the app, where you can see how it reuses most of the cached layers. This was the first rebuild which spent more time on those two layers noted above.

Rebuilding service "valuesaapi" after changes were detected...
[+] Building 15.2s (20/20) FINISHED docker:default
=> [valuesaapi internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.25kB 0.0s
=> [valuesaapi internal] load metadata for mcr.microsoft.com/dot 0.3s
=> [valuesaapi internal] load metadata for mcr.microsoft.com/dot 0.2s
=> [valuesaapi internal] load .dockerignore 0.0s
=> => transferring context: 383B 0.0s
=> [valuesaapi build 1/7] FROM mcr.microsoft.com/dotnet/sdk:8.0- 0.0s
=> [valuesaapi internal] load build context 0.0s
=> => transferring context: 1.74kB 0.0s
=> [valuesaapi base 1/4] FROM mcr.microsoft.com/dotnet/aspnet:8. 0.0s
=> CACHED [valuesaapi build 2/7] WORKDIR /src 0.0s
=> CACHED [valuesaapi build 3/7] COPY [API/ValuesTest.csproj, AP 0.0s
=> CACHED [valuesaapi build 4/7] RUN dotnet restore "API/ValuesT 0.0s
=> [valuesaapi build 5/7] COPY . . 0.1s
=> [valuesaapi build 6/7] WORKDIR /src/API 0.1s
=> [valuesaapi build 7/7] RUN dotnet build "ValuesTest.csproj" 11.0s
=> [valuesaapi publish 1/1] RUN dotnet publish "ValuesTest.cspro 3.4s
=> CACHED [valuesaapi base 2/4] WORKDIR /app 0.0s
=> CACHED [valuesaapi base 3/4] RUN apk add --no-cache icu-libs 0.0s
=> CACHED [valuesaapi base 4/4] RUN adduser -u 5678 --disabled-p 0.0s
=> CACHED [valuesaapi final 1/2] WORKDIR /app 0.0s
=> [valuesaapi final 2/2] COPY --from=publish /app/publish . 0.1s
=> [valuesaapi] exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:454f1258e665324777310e3df67674407b73e 0.0s
=> => naming to docker.io/library/api 0.0s
service "valuesaapi" successfully built

Also note that the pre-existence of the container may not be relevant. It is all about the build cache. So you may remove the container then run compose watch again and it will spin up very quickly. There are so many factors at play that as many times as you experiment, you may realize slightly different behavior. I have played with this many many times in order to come to a decent comprehension of what is going on (though not quite “how” …I’ll just chalk that up to Docker developer genius for now.)

But what if I do something like introduce a new package in my csproj file? (I’ll let you translate that question to the facets of your chosen coding stack.)

Adding a new Nuget package to the API’s csproj was, as expected, more expensive at about 50 seconds total. Here is the output from that change. Looking at the timing for each step, the bulk was restoring the newly added Nuget package over my SLOW internet! I could have run dotnet restore in advance, of course. Then it also required a rebuild of the project that could not rely on the cache.  That was over 13 seconds and another 3.7 to create the publish image.

Rebuilding service "valuesaapi" after changes were detected...
[+] Building 51.5s (20/20) FINISHED docker:default
=> [valuesaapi internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.25kB 0.0s
=> [valuesaapi internal] load metadata for mcr.microsoft.com/dot 0.5s
=> [valuesaapi internal] load metadata for mcr.microsoft.com/dot 0.5s
=> [valuesaapi internal] load .dockerignore 0.0s
=> => transferring context: 383B 0.0s
=> [valuesaapi build 1/7] FROM mcr.microsoft.com/dotnet/sdk:8.0- 0.0s
=> [valuesaapi base 1/4] FROM mcr.microsoft.com/dotnet/aspnet:8. 0.0s
=> [valuesaapi internal] load build context 0.0s
=> => transferring context: 1.47kB 0.0s
=> CACHED [valuesaapi build 2/7] WORKDIR /src 0.0s
=> [valuesaapi build 3/7] COPY [API/ValuesTest.csproj, API/] 0.0s
=> [valuesaapi build 4/7] RUN dotnet restore "API/ValuesTest.cs 32.8s
=> [valuesaapi build 5/7] COPY . . 0.1s
=> [valuesaapi build 6/7] WORKDIR /src/API 0.0s
=> [valuesaapi build 7/7] RUN dotnet build "ValuesTest.csproj" 13.5s
=> [valuesaapi publish 1/1] RUN dotnet publish "ValuesTest.cspro 3.7s
=> CACHED [valuesaapi base 2/4] WORKDIR /app 0.0s
=> CACHED [valuesaapi base 3/4] RUN apk add --no-cache icu-libs 0.0s
=> CACHED [valuesaapi base 4/4] RUN adduser -u 5678 --disabled-p 0.0s
=> CACHED [valuesaapi final 1/2] WORKDIR /app 0.0s
=> [valuesaapi final 2/2] COPY --from=publish /app/publish . 0.2s
=> [valuesaapi] exporting to image 0.3s
=> => exporting layers 0.3s
=> => writing image sha256:a5f16fef8705421b73499f2393cf7e790b958 0.0s
=> => naming to docker.io/library/api 0.0s
service "valuesaapi" successfully built

Removing the Nuget package forced another  expensive restore:

 => [valuesaapi build 4/7] RUN dotnet restore "API/ValuesTest.cs 10.0s

And re-building the layered images took about the same time as before.

=> [valuesaapi build 7/7] RUN dotnet build "ValuesTest.csproj" 11.1s
=> [valuesaapi publish 1/1] RUN dotnet publish "ValuesTest.cspro 3.4s

Lesson Number 2, compose syntax impacts your experience

With the develop/watch section in place in docker-compose, you can then trigger the watch with docker compose in two ways:

docker compose watch or

docker compose up –watch.

But the two ways have different effects and it took me some experimenting to understand the effects. Below I will lay out the various syntaxes for comparison so that you can avoid some of the confusion I encountered as I was learning.

compose up variations 

docker compose up

If you run docker compose up without the –watch flag, the Docker extension will give you the option to watch the source with the w key. If you do that, code changes trigger rebuild and you’ll get an updated API.

The CLI terminal (in my case, zsh) gets replaced with a docker terminal.

All info goes to this terminal including output debug / log info from dotnet.

If you enabled watch, then the rebuild details (as above) will also be displayed in this terminal window.

CTRL-C stops the containers, closes the docker terminal, returns to zsh terminal and prompt. Containers are still there, just stopped, not running.

If you want to remove the containers, you’ll have to use  docker compose down command or by right clicking the container in the docker extension explorer and choosing remove or right clicking the docker-compose file in the explorer and choosing compose down.

docker compose up -d or docker compose up -d watch

If you want Docker to watch, this is not your friend!

Running compose up in interactive mode, that is with the -d (detached) flag, will never work with watch. It won’t activate because Docker releases control back to you and therefore is not able to keep an eye on code changes. If you don’t add the –watch attribute, you will not get the prompt to start watch with the “w” key. If you do add the –watch attribute, watch will simply not activate. But there is no warning and was confused until I realized what was going on. Having learned this the hard way, I hope this will help you avoid the same confusion.

Another detriment to interactive/detached mode with or without the watch is that dotnet debug/log output is not displayed in the terminal. Or in any of the windows. You’ll have to force it to another target in your code.

Why so much focus on detached mode? In the Docker extension, if you right click a compose file and choose compose up from the extension’s context menu, the default command executed includes the -d. You can override those commands in the settings if you want.

In order to stop/remove containers ,you need to use the CLI or the tooling to trigger the CLI command e.g. docker compose down. CTRL-C will not have any effect. 

Using compose watch command instead of compose up command

 docker compose watch

Note there isn’t even a detached option available with watch because it’s totally irrelevant.

The terminal will tell you that watch is enabled when compose is finished spinning up the containers. As we are not interactive, you will not be released to a terminal prompt. Logs and message from the app will display in the Docker terminal as well as notifications from Docker. Code changes trigger rebuild (visible in terminal) and the app gets updated.

CTRL-C closes the Docker terminal and brings you back to a prompt and Docker stops watching. The containers are still running, however. But if you make code changes, they will not be updated because watch was no longer active. Now that the Docker terminal is gone, dotnet logs and other message coming from within the container are NOT output anywhere.

In this state (containers running, watch disabled, docker terminal gone and a recent change made to the code) I ran docker compose watch again from the prompt. Even though the containers were still running, keep in mind that the code change I made wasn’t handled. So the dotnet app had to be rebuilt and the API container updated. The dotnet build command took about 15 seconds and the entire composition was about 20 seconds.

Don’t Overlook compose watch!

After 30+ years coding, I still always tend to fear change in my tools .. worried that it will make me confused and feel stupid. (It usually does for a few minutes ..or perhaps longer…but then you figure it out It and feel like a million bucks!). It took me a long time before I looked at compose watch (with some encouragement) and in my typical fashion I had to try out what happens if I do this?  what happens if I do that? until I had a good idea of how its works and how incredibly useful it is! Don’t be shy…. it is an incredibly useful feature if you are developing with Docker.

 

 

 

Docker Init for ASP.NET Core Compared to VS or VS Code Extensions

Docker began introducing a new CLI command, docker init, allowing us to easily build stake-in-the-ground Dockerfile, docker-compose and .dockerignore files for a project. As the documentation says, these are “sensible files”. That is because init comes with templates to build these files based on your language.

docker init is a plug-in that, because it is a developer tool, is installed with Docker Desktop.

Early on, there were templates for HP, Python, Rust and other languages that I don’t code in. Finally the ASP.NET Core template arrived and it was time for me to start trying it out.

But I already had ways to create ASP.NET focused Dockerfiles and docker-compose files. The Visual Studio Code Docker extension allows you to add those to existing projects. Visual Studio can infer Dockerfile when you are creating a new project using a template or add that and compose after the fact.

Curious as to how these all differ, I explored the variations and will share those with you here. I will first do so with an ASP.NET Web API generated from a template and then open up an existing multi-project API that interacts with a database and see if that has any impact on the generated files.

JetBrains Rider also has Docker support but I did not include that in my already lengthy investigation. You can learn more about that support in their docs: https://www.jetbrains.com/help/rider/Docker_tools_for_net_projects.html#docker_support.

The solutions I created in this article are shared at https://github.com/julielerman/DockerInitComparison

Setting up Test 1: docker init with a dotnet new webapi

At the command line, in a new folder I’ve named dockerinittest, I’m just running dotnet new webapi.

This creates my little API, using the folder name as the default projet name.

Once that little bitty app is created, I then type in docker init. The command will gather a few bits of info and use what it has discovered in your folder to seed the responses which you can change.

  • App platform
  • Name of main project
  • Version of .NET
  • Local port which will default to 8080.

Here’s how that looks:

I love the “Don’t see something you need? Let us know!”. If you select that option, it will open up Google Form for you to share your ideas with dev team.

Notice that ASP.NET Core is already selected (notice the arrow to its left) because it was detected in the folder. So I just hit enter to get those sensible defaults created.

The tool then asks for the rest of the options with prompts based on my project:

? What application platform does your project use? ASP.NET Core

? What’s the name of your solution’s main project? DockerInitTest

? What version of .NET do you want to use? 8.0

After the final response, init then creates the sensible files (sorry, I just can’t resist) and provides a bit of guidance for running the API in Docker.

CREATED: .dockerignore
CREATED: Dockerfile
CREATED: compose.yaml
CREATED: README.Docker.md
✔ Your Docker files are ready!

Take a moment to review them and tailor them to your application.

When you’re ready, start your application by running: docker compose up –build

Your application will be available at http://localhost:8080

Consult README.Docker.md for more information about using the generated files.

Setting up Test 2: VS Code extension with a dotnet new webapi

For the second test, I have a new folder called VSCodeCreate. In there, I again use the dotnet cli to create a new web API project. Then I open that project in Visual Studio Code with “code .” at the command line which is the easy way in Windows.

I already have the Docker extension (created by Microsoft) installed in VS Code, so after opening the command palette with F1, I can choose “Docker: Add Docker files to workspace” . Like docker init, it presents me with some options.

  • Application platform: In my case .NET: ASP.NET Core is on the top because it was my most recently used option and I select that again.
  • Operating system: Windows or Linux. I choose Linux. I have no need for a Windows container.
  • In fact, it helpful asks: “What port(s) does your app listen on? Enter a comma-separated list, or empty for no exposed port.” Today it is defaulting to 5063 and I choose that.
  • Include optional Docker Compose file? I don’t really need this for the single project solution, but for the sake of comparison, I will say Yes.

That’s the end of the options and the extension creates a Dockerfile, a pair of compose files: docker-compose and docker-compose.debug, and a .dockerignore file. The debug file is really handy and it makes sense that VS Code can use that since it’s an IDE where you can debug your code. docker init , a command line tool, doesn’t have any guarantees that you’ll be debugging your code and therefore it didn’t really make sense for it to create such a file.

Setting up Test 3: Visual Studio with a new ASP.NET Web API

In Visual Studio, I use the ASP.NET Core Web API template to create a new project. The template’s wizard has prompts for enabling Docker and choosing which OS the container should be created for.

The Web API project is created with a Dockerfile and a .dockerignore file.

What about a docker-compose file? That gets added as a separate step. However, an interesting bug (https://github.com/MicrosoftDocs/visualstudio-docs/issues/9789) prevents me from adding it because my VSCreateFiles.sln file and VSCreateFiles.csproj are in the same folder! That’s what you get for creating demoware mindlessly.

I’ll restart and be sure to uncheck “place solution and project in same folder” in the wizard. I’m still letting it create the Dockerfile. With this, Container Orchestration support is an option. After specifying that it should target a Linux container, the compose assets are created.

That’s a comparison of the experience. Now let’s look at what gets created by each option.

A Handy Readme From docker init

docker init adds a Readme markdown file that provides some intro level guidance on running the app in the container, deploying the container, publishing the container and provides links to additional resources. Super helpful information.

Comparing the .dockerignore files

Next is the .dockerignore file. I’ve listed the contents of each in the table below. They all have the same core set of files which in fact is the entirety of what the VS Code extension has included. docker init has added one extra, .DS_Store which is short for Desktop Services Store, a file related to MacOS.  I created this on Windows, but the docker init is generic so it’s covering its bases. Visual Studio added an Azure specific yaml file (in the middle) as well as some additional git files (listed at the end). The outliers are in red.

Docker init dockerignore VS Code extension Visual Studio
**/.DS_Store
**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm**/bin
**/charts
**/docker-compose*
**/compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md

**/.classpath

**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm**/bin
**/charts
**/docker-compose*
**/compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md

**/.classpath

**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/azds.yaml
**/bin

**/charts
**/docker-compose***/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md
!**/.gitignore
!.git/HEAD
!.git/config
!.git/packed-refs
!.git/refs/heads/**

Comparing the Dockerfiles

Now let’s look at the Dockerfiles created for each variation.

Via docker init

Like the readme file, the Dockerfile created by docker init is filled with instructional comments describing what each line of code is for. It also provides hints of common changes you might want to make and why and links to more information. Removing the comments, here is the actual code in the file.

FROM –platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS build
ARG TARGETARCH
COPY . /source
WORKDIR /source
RUN –mount=type=cache,id=nuget,
target=/root/.nuget/packages \

  dotnet publish -a ${TARGETARCH/amd64/x64}
–use-current-runtime

self-contained false -o /app
FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine AS final
WORKDIR /app
USER $APP_UID
ENTRYPOINT [“dotnet”, “DockerInitTest.dll”]

This is doing a staged build of the image. Starting with a base of the dotnet SDK(first FROM), copying in the source and compiling the application (RUN).

Then it takes that compiled application and copies it into a smaller image that’s based on only the ASPNET core runtime (second FROM) as it no longer requires the larger SDK to build. With that image, it finally defines a user and entry point for when you build a container from this image.

Via VS Code Extension

Now let’s see what the Docker extension for VS Code built. Theres a lot more code here and no comments.

I remember being overwhelmed years ago the first time I saw what this extension created. But the ASPNET docs provide a detailed explanation, which I had to read through numerous times, and so my learning journey began.

This, too, creates a staged build named “build”. But while the previous Dockerfile built two stages, this one uses four. The first is that it’s creating an image (“base”) from the aspne core image and setting it aside. Then like above, it creates an image based on the SDK and pulls in the needed files. The steps here are more explicit about file names than the previous Dockerfile. Next it explicitly restores the Nuget packages, rather than just letting that happen during the build.  If you look closely at the complex RUN command from the first Dockerfile, you’ll see that it is compressing these steps into that single command.

The third stage publishes the application into a new image (“publish”) from the SDK-based “build” image and publishes the file into a folder on this new image. Again, notice that publish is part of the RUN command from the docker init generated file.

Finally, we return to the set-aside “base” image and create our fourth and final image, named “final”, into which the published application files are copied. This is the image that containers will be built from and therefore defines the entry point.

One other point is that this Dockerfile specifies the port (5063) on which the app will be exposed by any container created from the image.

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 5063

ENV ASPNETCORE_URLS=http://+:5063

USER app
FROM –platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0
AS build
ARG configuration=Release
WORKDIR /src
COPY [“VSCodeTest.csproj”, “./”]
RUN dotnet restore “VSCodeTest.csproj”
COPY . .
WORKDIR “/src/.”
RUN dotnet build “VSCodeTest.csproj” -c $configuration -o /app/build

FROM build AS publish
ARG configuration=Release
RUN dotnet publish “VSCodeTest.csproj” -c $configuration -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY –from=publish /app/publish .
ENTRYPOINT [“dotnet”, “VSCodeTest.dll”]

Whew!

Via Visual Studio

Now for the Dockerfile created by Visual Studio as part of the ASP.NET Core project template. It is nearly the same as the one created by the VS Code extension. This isn’t a surprise to me as they are both defined by Microsoft and likely by teams that are collaborating. I won’t bother listing the entire thing.

The most significant difference is that it exposes two ports, 8080 and 8081, the latter to be used as the https target.

EXPOSE 8080
EXPOSE 8081

In the long run, the Visual Studio Code and VS generated Dockerfiles are clearer to read but achieve the same result as the docker init version which is minimal image that contains the app and is not bloated with the SDK. docker init does it in fewer passes. On my system, any differences between the resources used and time taken were not noticeable although I did not perform any serious benchmarking.

Comparing the Compose Files

This is where you’ll find the most significant differences between the three.

As a reminder, docker init and VS Code’s Docker extension create the docker compose file along with Dockerfile and VS requires you to explicitly create compose.

Consider the fact that in all three cases, I have only a single project and am not involving a database or other resources that might require additional images. In all three cases, compose isn’t really necessary as I have no need to orchestrate multiple images. docker run using the single image will work.

docker init’s compose

The docker init generated file is simply named compose.yml and its actual code is simple as well. It will create one container named services based on the image named final generated by Dockerfile. And it will expose  the service on the host computer’s port 8080 and internally on the container’s port 8080.

services:
  server:
    build:
      context: .
      target: final
    ports:
      – 8080:8080

But I referred to this as the “actual code”. That’s because the file is filled with comments that provide further guidance and links to documentation, especially helpful for those new to Docker. And it also provides a commented example of running a second container to supply a PostgreSQL database via the official postgres image. That is very handy because otherwise it’s off to google, or github.com/docker/awesome-compose or other existing projects to copy and paste the same. I also would like to say that with this project open in VS Code, I tested out GitHub Copilot in the yaml file.

Here is my prompt:

#create a service using Microsoft SQL server docker image

And here is what it generated, ignoring the fact that the server service already exists, but also very likely having read the code already in the file:

services:
  server:
    build:
      context: .
      target: final
    ports:
      – 8080:8080
  db:
    image: mcr.microsoft.com/mssql/server
    restart: always
    environment:
      – ACCEPT_EULA=Y
      – SA_PASSWORD=YourStrongPassword
    ports:
      – 1433:1433

 

This made me very happy, knowing what I went through back in my early Docker days to comprehend and achieve the same.

The original example in the yaml file for postgres is more extensive and you can see it in my GitHub repo.

VS Code’s Compose

Now let’s take a look at what the VS Code extension created.

Recall that above I said that this created two files: docker-compose.yml and docker-compose-debug.yml.

The docker-compose.yml file does have one comment which I’ll include the full listing:

# Please refer https://aka.ms/HTTPSinContainer on how to setup
# an https developer certificate for your ASP.NET Core service.
version: ‘3.4’
services:
  vscodetest:
    image: vscodetest
    build:
      context: .
      dockerfile: ./Dockerfile
    ports:
      – 5063:5063

This is a bit more explicit than what docker init created. It specifies the version and names the container using the name of the project and image. Also, rather than specifying it should build form a particular image, it says to leverage the contents of the Dockerfile to learn what it needs. Finally, it exposes both ports on 5063 as per the Dockerfile specification.

Interestingly, the same GitHub Copilot prompt results in the following in this file:

 #create a service using Microsoft SQL server docker image
    sqlserver:
      image: “mcr.microsoft.com/mssql/server:2019-latest”
      environment:
        SA_PASSWORD: “Password123”    

This is missing some important details like the ACCEPT_EULA attribute and ports for the image.

I know this because I’m not relying on Copilot for 100% knowledge, but just to do some quick typing for me. Especially for the URL of the image! Who memorizes that stuff?

I cannot explain why it is so different from what it generated in the other project although someone with more expertise with Copilot might be able to.

The second compose file allows you to debug the code when it’s running in the container which is a glorious capability in VS Code. VS can do it, too. It uses the same image created from the Dockerfile but applies some additional parameters. It also uses a volume which, from my research, is where the source code remains accessible to the debugger. However, I could not see evidence of a volume in either the Docker Explorer in VS Code or in Docker Desktop. Again, brighter minds than mine might be able to explain.

# Please refer https://aka.ms/HTTPSinContainer on how to setup an https developer certificate for your ASP.NET Core service.
version: ‘3.4’
services:
  vscodetest:
    image: vscodetest
    build:
      context: .
      dockerfile: ./Dockerfile
      args:
        – configuration=Debug
    ports:
      – 5063:5063
    environment:
      – ASPNETCORE_ENVIRONMENT=Development
    volumes:
      – ~/.vsdbg:/remote_debugger:rw
With or without that volume showing itself to me, running compose up on that yml file does indeed allow me to set breakpoints and perform all the normal debugging tasks even though it’s all executing inside a container.

Visual Studio’s Compose

Now let’s turn our attention to the compose file created in Visual Studio using the “Container Orchestration support” menu option.

First of all, it’s important to know that the compose files are in the solution but not in the project which makes perfect sense. In the other scenarios, everything was contained in the single folder for the project. If you added more folders, it is up to you to move the compose files outside of the folders.

Here is how that looks in the Solution Explorer:

The heading is a special type of project file with a dcproj extension, whose contents describe the files included and various attributes about the container. It’s interesting and I recommend looking at it in the repository.

The compose file is a bit different than the one created by the VS Code extension:

  • It does not specify ports.
  • It will dynamically find the vscreatefiles image
  • It knows that the Dockerfile is in a different folder.

version: ‘3.4’

services:
  vscreatefiles:
    image: ${DOCKER_REGISTRY-}vscreatefiles
    build:
      context: .
      dockerfile: VSCreateFiles/Dockerfile

Notice that there is no explicit debug compose file. Instead, the wizard created an additional configuration in launchsetting.json for running or debugging via Docker Compose and that is one of the options for running with or without debugging, i.e. CTRL-F5 or just F5.

Conclusions

I have definitely sated my curiosity and feel that each of these options are equally sufficient for whichever development environment you are working in. That is, in my opinion, the key factor. Are you working at the command line or one of the IDEs? They will each give you a good, working head start.

docker init is going to be much more useful for people who just dive in to Docker with little advance knowledge because there is so much guidance embedded in the comments. The support for Visual Studio and VS Code also provide enough to get your app running but will require some help from Copilot or Dr. Google to satisfy more complex solutions and needs.

 

 

 

 

 

 

 

 

 

A Small Lesson on env Files in docker-compose

I’ve been working a lot with docker lately and learning learning learning. I have written a 3-part series for my MSDN Mag Data Points column that will be out in April, May and June 2019 issues. I have another YUGE blog post that I will publish to accompany the May article. And I’m working on others.

I explored Docker environment variables and different ways to feed them into a Docker image or container.

My docker-compose file referenced an environment variable named DB_PW without specifying it’s value.

dataapidocker:
image: ${DOCKER_REGISTRY-}dataapidocker
build:
context: .
dockerfile: DataAPIDocker/Dockerfile
environment:
- DB_PW

Docker will read environment variables from the host to discover the value.

But this was a pain. I wasn’t permanently storing the DB_PW on my development machine and had to remember to set it frequently. Elton Stoneman (from Docker) said I should *really* consider using the Docker env file feature. This lets you set variables in an environment file and let the docker-compose file read from that. And I can keep that special file out of my source control. (Always my worry!)

I started by following docs that showed using a file named anything.env. I created a file called hush-hush.env where I specified the variable. This is the full content of the file:

DB_PW=thebigsecret

Then in docker-compose, in the service, the env_file tag lets you point to it. You can even remove the environment tag in the yml file.

dataapidocker:
image: ${DOCKER_REGISTRY-}dataapidocker
build:
context: .
dockerfile: DataAPIDocker/Dockerfile
env_file:
- hush-hush.env

This worked perfectly. My app was able to discover the environment variable in code.

But then I evolved my solution to use another container for my database. The primary container depends on the new mssql container. And the mssql container requires I pass in the database password as an environment variable. Since DB_PW already exists, I can do that easily enough with substitution (via curly braces). Here’s the new docker-compose file:

version: '3.4'
services:
dataapidocker:
image: ${DOCKER_REGISTRY-}dataapidocker
build:
context: .
dockerfile: DataAPIDocker/Dockerfile
env_file:
- hush-hush.env
depends_on:
- db
db:
image: mcr.microsoft.com/mssql/server
volumes:
- mssql-server-julie-data:/var/opt/mssql/data
environment:
SA_PASSWORD: "${DB_PW}"
ACCEPT_EULA: "Y"
ports:
- "1433:1433"
volumes:
mssql-server-julie-data: {}

And there’s an order of operations issue here. When I build the docker-compose file, it complains that DB_PW is not available and my app fails. The db service is not getting the contents of my hush-hush.env file. I tried a number of things, such as adding env_file to the db service. In the end, here’s what I learned.

The substitution use requires that the DB_PW variable be defined in docker-compose. I added that back in to the primary service, but it was not getting the value from hush-hush.env.

But you can have a .env file that has no name. The extension *is* the full name of the file. Docker-compose will read that early enough and provide the value from the .env file to the declared DB_PW. Then all of the pieces fell in place. The mssql container was spun up with the value from DB_PW as its environment variable. And my app code was able to read the environment variable that Docker passed into the running container for its own tasks.

The final docker-compose.yml file looks like this:

version: '3.4'
services:
dataapidocker:
image: ${DOCKER_REGISTRY-}dataapidocker
build:
context: .
dockerfile: DataAPIDocker/Dockerfile
environment :
- DB_PW
depends_on:
- db
db:
image: mcr.microsoft.com/mssql/server
volumes:
- mssql-server-julie-data:/var/opt/mssql/data
environment:
SA_PASSWORD: "${DB_PW}"
ACCEPT_EULA: "Y"
ports:
- "1433:1433"
volumes:
mssql-server-julie-data: {}

And it relies on a file named “.env” with my variable key value pair defined (same as hush-hush.env above).

DB_PW=thebigsecret

Resources:
https://docs.docker.com/compose/environment-variables/

Getting the SQL Server 2019 for Linux CTP2.0 Docker Image

If you are used to pulling the mssql-server images from the microsoft repository, e.g.,

docker pull microsoft/mssql-server

that won’t work for the 2019 CTP.

I was able to repull (aka update) using the former repository, but that wasn’t working for the CTP whose tag is vNext-CTP2.0-ubuntu.

I finally noticed the new docker pull command on the docker hub page for the image

It says: docker pull mcr.microsoft.com/mssql/server

So the command for pulling the CTP using it’s tag is as follows:

docker pull mcr.microsoft.com/mssql/server:vNext-CTP2.0-ubuntu

 

What’s in that sqlservr.sh file on the mssql-sqlserver-linux docker image anyway?

Update June 3, 2017: The team has revised the docker image and the bash file is gone, presumably with its logic broken up in to various locations. Still I’m glad I grabbed this when I did to satisfy my curiosity!

Microsoft has created 4 official Docker images for SQL Server: SQL Server for Linux, SQL Server Developer Edition, SQL Server Express and (windows) SQL Server vNext) . They can be found on the Docker hub (e.g. https://hub.docker.com/r/microsoft/mssql-server-linux/) and there is also a Github repository for them at github.com/Microsoft/mssql-docker. Some of the files that go along with that image are not on Github. The Dockerfile files for each image run some type of startup script. The Windows images have a PowerShell script called start.ps1. You can see those in the Github repo. The Linux image runs a bash file called sqlservr.sh. That’s not included in the repo though and I was curious what it did.

Note: I wrote a blog post about using the SQL Server for Linux container (Mashup: SQL Server on Linux in Docker on a Mac with Visual Studio Code and I’m also writing an article about using the containers for my July MSDN Magazine Data Points column (watch this space).

Still a bit of a bash noob, I learned how to read a file from a docker container on ..you guessed it…StackOverflow.  Following those instructions, I created a snapshot of my running container

MySqlServerLinuImage git:(master) docker commit juliesqllinux  mysnapshot

sha256:9b552a1e24df7652af0c6c265ae5e2d7cb7832586c431d4b480c30663ab713f0

and ran the snapshot with bash:

  MySqlServerLinuImage git:(master) docker run -t -i mysnapshot bin/bash

root@2a6f950face2:/# 

Then at the new prompt (#), used ls to get the listing

root@2a6f950face2:/# ls

SqlCmdScript.sql  SqlCmdStartup.sh  bin  boot  dev  entrypoint.sh  etc  home  install.sh  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

then navigated to  folder where the bash file is and listed its contents:

root@2a6f950face2:/opt/mssql/bin# ls

compress-dump.sh  generate-core.sh  mssql-conf  paldumper  sqlpackage  sqlservr  sqlservr.sh

Once I was there I used the cat command to list out the contents of the sqlservr.sh file and see what it does. Here is the secret sauce in case, like me, you NEED to know what’s going on under the covers!

root@2a6f950face2:/opt/mssql/bin# cat sqlservr.sh 

#!/bin/bash

#

# Microsoft(R) SQL Server(R) launch script for Docker

#

ACCEPT_EULA=${ACCEPT_EULA:-}

SA_PASSWORD=${SA_PASSWORD:-}

#COLLATION=${COLLATION:-SQL_Latin1_General_CP1_CI_AS}

have_sa_password=""

#have_collation=""

sqlservr_setup_prefix=""

configure=""

reconfigure=""

# Check system memory

#

let system_memory="$(awk '/MemTotal/ {print $2}' /proc/meminfo) / 1024"

if [ $system_memory -lt 3250 ]; then

    echo "ERROR: This machine must have at least 3.25 gigabytes of memory to install Microsoft(R) SQL Server(R)."

    exit 1

fi

# Create system directories

#

mkdir -p /var/opt/mssql/data

mkdir -p /var/opt/mssql/etc

mkdir -p /var/opt/mssql/log

# Check the EULA

#

if [ "$ACCEPT_EULA" != "Y" ] && [ "$ACCEPT_EULA" != "y" ]; then

 echo "ERROR: You must accept the End User License Agreement before this container" > /dev/stderr

 echo "can start. The End User License Agreement can be found at " > /dev/stderr

 echo "http://go.microsoft.com/fwlink/?LinkId=746388." > /dev/stderr

 echo ""

 echo "Set the environment variable ACCEPT_EULA to 'Y' if you accept the agreement." > /dev/stderr

 exit 1

fi

# Configure SQL engine

#

if [ ! -f /var/opt/mssql/data/master.mdf ]; then

 configure=1

 if [ ! -z "$SA_PASSWORD" ] || [ -f /var/opt/mssql/etc/sa_password ]; then

 have_sa_password=1

 fi

# if [ ! -z "$COLLATION" ] || [ -f /var/opt/mssql/etc/collation ]; then

# have_collation=1

# fi

 if [ -z "$have_sa_password" ]; then

        echo "ERROR: The system administrator password is not configured. You can set the" > /dev/stderr

        echo "password via environment variable (SA_PASSWORD) or configuration file" > /dev/stderr

        echo "(/var/opt/mssql/etc/sa_password)." > /dev/stderr

 exit 1

 fi

fi

# If user wants to reconfigure, set reconfigure flag

#

if [ -f /var/opt/mssql/etc/reconfigure ]; then

 reconfigure=1

fi

# If we need to configure or reconfigure, run through configuration

# logic

#

if [ "$configure" == "1" ] || [ "$reconfigure" == "1" ]; then

 sqlservr_setup_options=""

# if [ -f /var/opt/mssql/etc/collation ]; then

# sqlservr_setup_options+="-q $(cat /var/opt/mssql/etc/collation)"

# else

# if [ ! -z "$COLLATION" ]; then

# sqlservr_setup_options+="-q $COLLATION "

# fi

# fi

 set +e

 cd /var/opt/mssql

 echo 'Configuring Microsoft(R) SQL Server(R)...'

 if [ -f /var/opt/mssql/etc/sa_password ]; then

 SQLSERVR_SA_PASSWORD_FILE=/var/opt/mssql/etc/sa_password /opt/mssql/bin/sqlservr --setup $sqlservr_setup_options 2>&1 > /var/opt/mssql/log/setup-$(date +%Y%m%d-%H%M%S).log

 elif [ ! -z "$SA_PASSWORD" ]; then

 SQLSERVR_SA_PASSWORD_FILE=<(echo -n "$SA_PASSWORD") /opt/mssql/bin/sqlservr --setup $sqlservr_setup_options 2>&1 > /var/opt/mssql/log/setup-$(date +%Y%m%d-%H%M%S).log

 else

 if [ ! -z '$sqlservr_setup_options' ]; then

 /opt/mssql/bin/sqlservr --setup $sqlservr_setup_options 2>&1 > /var/opt/mssql/log/setup-$(date +%Y%m%d-%H%M%S).log

 fi

 fi

 retcode=$?

 if [ $retcode != 0 ]; then

 echo "Microsoft(R) SQL Server(R) setup failed with error code $retcode. Please check the setup log in /var/opt/mssql/log for more information." > /dev/stderr

 exit 1

 fi

 set -e

 rm -f /var/opt/mssql/etc/reconfigure

 rm -f /var/opt/mssql/etc/sa_password

 echo "Configuration complete."

fi

# Start SQL Server

#

exec /opt/mssql/bin/sqlservr $*