Food, code, & else

Deployment Branches in Azure

 

If you’re familiar with Azure, you’ll recognize immediately that everything is setup to be fairly flat. That’s mostly by design, as the whole point of any cloud resource was to be as scalable as possible into giant monstrosities as quickly as possible. What it wasn’t meant to be was a place to create branching strategies with easy to recreate environments. Usually what many developers do is create VMs instances of the things they want. For example, make a VM with a SQL server instance installed, not an Azure SQL database that sits on it’s own. If that was your situation, creating a duplicate environment would be as simple as duplicating that VM.

But what if that wasn’t the case? What if you had a serverless environment with all of these floating entities instead? How would you create a dev, prod, or test version of these things? We had a client that had this exact situation in an enterprise situation, and had to put together something that got us everything we wanted. There are a couple of ways to do it (some better than others). What I’m going to do here today is talk about how we ended up making that happen. Is it perfect? Definitely not (I’ll talk more on this later), but it got us to where we needed to be. “What about deployment slots?”, you experts just asked. I mention that later, too.

A Background: Standard Deployment Branches

If you’re someone familiar with programming for enterprises or best practices, you’re most likely familiar (or have heard of) branching. Especially if you’re ever worked with git before for version control. If you aren’t familiar with these concepts, here is a great run down about workflows from Atlassian. I’ll do a quick overview here, but since it also is a very fundamental concept I highly suggest reading up on it.

The 30 seconds or less version of that Atlassian tutorial goes like this. Let’s say you have a piece of code that you want to develop on for a website, say Kat’s Ice Cream’s website. You have the live website where anyone can find out about Kat’s Ice Cream (the newest flavor is salted candied bacon and caramel), see their hours, location, and more. But what happens when you want to change something on that website, say a new browser application that lets you order ice cream to go (for a fee, of course)? You currently only have that one website, so you would have to test your new application on live customers, bugs and all. How does that sound?

 

Terrible! Imagine having to deal with thousands of hungry ice cream eaters (Kat’s Ice Cream is incredibly popular) not getting their ice cream needs filled? Could you imagine the poor cashier having to sit there giving website help over the phone when they should be handing out delicious ice cream for the populous?

The solution? Branches & dedicated environments. You create a development, test, and production branch for the website. Each one of these are sister environments with the exact same website set up as what your end production system will be. The development code branch is your sandbox, where you can throw all of your crazy ideas at it and see what will stick. Your testing is a twin to production, allowing any QA team to test and hit it. And your production (or master if you’re a git person) is your final and polished product. Now if you wanted to create a new big feature, and not effect anyone else’s code, you would create a smaller branch of the dev environment, create and iron out your feature, then merge it back up to development whenever you’re ready for your code to be deployed. You’ll probably see a visual that maps to that and looks something like this:

I took this from Atlassian’s tutorial I mentioned earlier. Can’t recommend reading it enough.

Branching in Azure

I’m going to step through the system here, but at the end of the day, you’ll need these things from Azure in order to have a branchable and reproducible system:

  • ARM Templates
  • Build Definitions
  • Config Transforms

Best Practices

But, before we go into detail, I want to mention a few standards you should follow, and that I’m using here in this system.

 

Resource Groups & Subscriptions

I have one resource group for each of my branches, and they will be duplicates of each other in other ways. For example, I’ll call one KatsIceCreamSite-Dev versus KatsIceCreamSite-Test for a Dev and Test branch. Azure does not let you right click and copy instances, or duplicate into a branch. Our solution was to create a resource group per “environment”. Azure doesn’t really have environments you can create, but they do have billing subscriptions. If you can afford a subscription per environment as well (i.e. a test subscription, a dev, and so forth) I would recommend it. It makes the barrier between environments a lot easier, and you can control permissions for them as well.

 

Naming

I will denote any instances for development or prod with a “-Dev” or “-Prod”. When you have a 500+ instance Azure subscription, it can save you some trouble if you have a clear knowledge of what you’re doing on what “environment”.

 

Deploying Code

Should we ever build code and deploy it straight to production?

Nope.

Why not? Because when you develop code you make mistakes. Technically only one person or team should have access to deploy to production, and that’s only after good QA testing and the like. You never want to deploy something to the web site that could go live. We will, however, make a build definition for production that you can queue a build of.

 

Let’s get started.

Our system

First thing first, let’s talk about what our example system has. Say the ice cream shop has a database for orders and flavors, an API app to expose the data to the DB, and a web application hosting the website for this system. The web app is maybe a .NET based website using angularjs, the API app is your standard API using C#, and the DB has a table for flavors, orders, and other information we might need. It would look similar to this:

By using this design, we have everyone doing one job and only one job. The website just takes orders and visualizes information. The api app’s job is to record and expose data. And the DB’s job is to save that data. No VMs required.

Our Goal

What we want is the ability to create this same system, but with a production and development version. It must be automated in order for me to be able to create a duplicate environment of our master (using some git terms here) branch. We want only our most stable version to be exposed to the user. Therefore, your Azure instance should look like this:

Each of these branches will be represented by a separate resource group. In our case KatsIceCream-Dev and KatsIceCream-Prod.

How do we go from one instance of everything to multiple in an automated fashion? Some people would think, “well, just make a second instance of it”. In azure you can create two of one instance of something, like a web application, under the same name and URL (scaling out horizontally versus scaling up vertically). That’s not what we want here. we want a separate URL and twin instance far away from the production version. And what if we want to send this code out to other programmers to create on their own for testing or examples. How would they get this setup in Azure? We already know how to transfer code bases pretty well, but how can we transfer instances of Azure?

Well, that’s where ARM Templates step in.

ARM Templates

Azure has something called ARM Templates, short for Azure Resource Manager templates. ARM is an Azure tool that will create or remove instances of items. For example, I could tell ARM to create 2 databases and a service bus, and it could do that. But you should note, it will create empty instances of Azure resources. Zero code or tables. If you wanted to create a web application and deploy code, you would create separate steps of a build definition to do each of these things (something we’ll talk more in depth later). For now, know we’re making a blank instance of everything.

A template is basically a fancy JSON that Azure will run a powershell script on top of. You’ll give it things like names, location of your resources, resource groups to put those things in, and more. So, let’s make a ARM template for Kat’s Ice Cream.

If you’re looking for a tutorial on how to make an ARM template, the Azure documentation is very well versed. You also can manually create the instances in the portal and then export the ARM template from the portal, but I wouldn’t recommend it. It usually adds junk in there that you don’t really need, and therefore you will have one messy JSON. But here’s the steps I went through if you want to follow along.

Let’s create a solution for our template:

Feel free to tell me how wonderful my cat drawing is. I’ll wait.

When the wizard asks, select Blank Template.

You’ll notice three things in your ARM template’s solution explorer:

  • A powershell script
    • This would run the code locally if you wanted to
  • A parameter file
    • This holds the values you would plug into your template
  • A template file
    • This tells ARM what resources you want to make or update in Azure

If you were deploying and publishing from Visual Studio, I would tell you that the power shell script matters. Since we are using build definitions and gated check-ins, you can ignore it. Azure will run a similar script within the build definitions when we deploy using gated check-ins later. And, even if you go around it, I’ve had so many unexplained errors from deploying locally using that power shell script that I simply do not trust it. I would run it once, get an error, run it again, get none.

¯\_(ツ)_/¯

Run it through visual studio online, and it should work.

At this point I have copied my parameter file into two files, and renamed them. I want one for DEV for my development branch, and one for PROD, my production, exposed branch. I want the ARM template to have one base code of a JSON, and it can switch parameters if I want to deploy to dev or prod. If I ever wanted to redeploy to a new branch, I would just have to create a new parameter file and plug it in. Then I’m ready to go.

Let’s say I wanted to create a test environment. If I wanted another twin to do that, I would copy the parameters file for another one and name it with TEST instead. You would just point to the parameter file you’d like to use, and then ARM would produce the resources for that.

Now, let’s make some resources here. I’m going to make two web apps (one for the API, one for the site), and the SQL table. We’re using an app service plan for the two web applications to use for billing and location. A service plan is Azure’s way of grouping app services onto one billing group. You can do that with any Azure app services (functions, web apps, logic apps, etc). It’s required, but logically you don’t really need to know much more for that in order to understand branching.

To add something to a resource, you can right click the “resources” node in the tree on the JSON viewer, or follow the Azure ARM tutorials. You also can get the JSON and just paste it into the code if you feel comfortable there. It will pop up on the viewer if the syntax is correct.

One note, I have a huge pet peeve about Azure is that there are naming requirements not are not consistent across the board. For example, SQL requires all lowercase, but web apps can be any case, and some can be any case but no special characters. It’s a giant pain in the neck for naming conventions to try and find the shaded area of a 10 circle venn diagram.

 

Here’s the ARM template for the interested:

 

If you look at the code above, you’ll notice that the parameter values are injected. These are linked up to the parameter files we’ll fill in next. That’s where we can place our changing values for our instances. The wizard will probably have you use the variables, but I find the parameters to be much more useful.

In our parameters file, you can put the values that you want to use. You also can link up to other services, and when you deploy the Azure you can put in values from those services. For example, using Key Vault. You can put in connection strings or passwords as a secret in Key Vault, and then just link to the key vault with the name of the secret. It’s explained well here. I highly recommend the key vault! It’s more secure, and you also can make it as one of your outputs of your ARM. You technically could send that as an application setting to an instance of a web app through your ARM template, and then technically you never would have a connection string in the App.Config of a web application. Lots of great ideas here.

 

Here’s what it would like in our example parameter file:

Once our code is set, we can deploy this through Visual Studio via the right click publish (which is really wonky, wouldn’t recommend), using the Azure CLI, or using a build definition to deploy it through TFS.

Let’s talk Build Definitions.

Build Definitions

Build Definitions is a way for Team Foundation Server to have gated check ins for your code, and do custom steps whenever you check in your code. Microsoft actually setup a bit of a market place for add-ons in this sections for things like Jenkins or Slack, and you can develop them yourselves. But, for our purposes, all we want to do when we build is to the solution of our code, test it, and deploy to whatever resource group or subscription we need to. For example, we can build the API code, and deploy it to the development branch and/or resource group.

ARM Build Definitions

Let’s make a Build Definition for ARM. We’ll make it so that whenever we check in the code, it will update the Azure resources or instances that your ARM template now allocates. If you haven’t made a build definition before, check out this link. It’s pretty straight forward. But if I want to build and deploy an ARM template, I’ll have to use the Azure Resource Group step. The naming indicates that we will be uploading to a resource group, or creating a resource group to deploy to.

If you notice that this picture looks different, I’m using the latest version of the UI for visual studio online (because I’m cool like that). The tasks should be the same.

You’ll link up your task to point to the location of your checked in code for your ARM template, your subscription for Azure, and the resource group your code should be deployed to. Since this is for a dev branch, I want to setup a gated check in here later. Point it to your ARM project and template file. You also will select the dev or prod instance of the parameter file you will use. Since this one was dev, I pointed it to my DEV parameter file.

“Gotchas” to be aware of:

  • Resource Group Name
    • If you misspell the resource group here when you’re trying to refer to an existing one, it will create a new resource group under that misspelling. And it will deploy all these instances you created in your ARM. I hope you didn’t select a premium priced SQL!
  • Your Location
    • Unless you set the location in your ARM template to be inherited from your resource group, you technically can have a different site for your app than your resource group. You also will be charged differently if Azure has to reach out to different places. $$$

Add any other steps to your definition that you might need. For example, having a step to build your .sln. Save your build definition and setup your gated check in (under the Trigger tab) if you’re a build for dev code. Do not do gated check for your prod build definition.

Name your build definition KatsIceCreamARM-Dev since this is a build definition.

Repeat the process to get a second build definition for prod by selecting a prod resource group and prod template. Everything else should be the same. At the end of the day you should have two build definitions. You can see this in your Explorer view like so:

KatsIceCreamARM-Dev

KatsIceCreamARM-Prod

 

To run or start a build, you’ll queue a new build to run through the steps you setup here in your definition. If you have a gated check in, then it will queue a new build automatically whenever you check in. You can do so by clicking “queue new build”.

 

You should see this window. If you want to run and create those items in your ARM template, you can click “ok” here and run it.

 

Now you’ll notice some Variables here. Could I have technically put inputs here for both dev and prod and have it inserted into the build definitions what template to use? Yes, and I could have just have one build definition instead of two. However, I don’t like the idea of having a bad key stroke be the difference between deploying to your live instance and your dev instance. You could build to prod from your visual studio environment when you do a gated check in, which I do not recommend.

Once your build definition runs (you should see a screen that looks like an msbuild screen since, funnily enough, this is really a fancy wrapper over msbuild), your Azure instance will now have that SQL database and web applications in your portal for you to see.

Code Build Definitions

Let’s say you have a .NET (or java, or js, or whatever) solution for your two web applications all coded and checked in. What we want to do now is automate your code changes to be continuously integrated (CI), and deploy the code to your dev instance anytime you checked into TFS. Remember, the ARM template creates blank instances of stuff, but no code.

It’s going to be pretty similar to the ARM one. Have a .sln build or NuGet package restore step as you need. Now we’ll add a step to deploy. It’ll be different based on what you’re doing, but here’s what it looks like for a web app.

 

You can add the step, and then fill in the details as need be to point to your code’s paths for your dev environment. The rest of the system is pretty self explanatory, so you can save it here and create a prod definition as a clone. When you make your prod build definition, point it to your prod resource group and subscriptions. Double check that, as there are no restrictions in visual studio online for what your write here (I’ve found out this the hard way).

Now you should have four build definitions. Two for ARM (dev & prod) and two for code (dev & prod). You’re now set here.

Why shouldn’t I combine the ARM & Code definitions into one?

Technically you could have a build definition that does what the ARM template does and deploy the code at the same time. But, I don’t suggest it. My main argument is that if you had a gated check in, which path would you use? The ARM or code path? And, if you make a change in the ARM and a change in the code, then something goes wrong, what’s the cause of the bug? You don’t really know. I would recommend a definition for one deployment purpose. In my enterprise system, I have one for the web application, one for my storage items, one for my web jobs, etc. It keeps it simple, and lets you only make changes in the section you are affecting.

 

Config Transforms

Now, you’ll have different instances and URLs for dev and prod of your resources in Azure. Two keep the branches duplicate, you should have one code source with a configuration that points to prod and dev instances. How would you transform the code to hit the different items with different connection strings? You’ll need to do config transforms for app.configs files (web configs already are setup in Azure).

I’ve already written a post here on how to do application config transforms. You can follow the steps there to get set.

Now you’re done!

You now have an automated system that lets you create new instances of Azure resources in a parameter based way.

My Wish List & The Glass is Half Empty

Now, the fun part. What about this system stinks and can do better? Well, here is my personal wish list.

Missing ARM Support (Shoulder-less)

Not all resources in Azure are able to be deployed by an ARM template (as of the writing of this blog). For example, stream analytics. I can’t make stream analytics via an ARM template. Our work around? We saved the query in a text doc that we keep on source control, and try to keep it in sync with what we have on the portal. It’s regularly checked in by Hacky McHackerson.

Grouped Build Definitions

In an ideal world, I would like to be able to group these definitions into one pile and say to visual studio online “build all dev branch builds” in one go. Right now, all you can do is click individually each build definition one by one. Now, in theory, you could create a list that you ran via a powershell script and run it from there, but with a GUI that is so built out on TFS, you’d think that this would be a native built in function. And, what if I want all the dev builds to fail if one failed? I cannot set that up.

The only solution I’ve found is a add-on that is a step on a build definition to queue other builds. I.e., you would have one build definition to rule them all (keep it secret, keep it safe) that had a step to queue other builds. It isn’t everything I want, but maybe I’ll make my own one day with everything I want.

I feel like this should be a native built in feature, but they didn’t ask me.

Build Definition “Templates” and Restrictions

I’ve run into so many issues with simple, small mistakes with these. For example, when I’ve accidentally typed into an ARM build definition that I wanted KatsIceCream-Dev when it was a prod definition in the name, I’ve posted to the wrong resource group. I then had to manually go into the portal and delete that instance and queue a new build again. It’s very easy for one keystroke to make a $400 mistake if you don’t notice it and let it run. In an ideal world, I would be able to setup a prod and dev build “template” so that anyone creating one of these items has to have certain restrictions. Say, a certain naming convention or to a particular subscription, etc. That would save a lot of heartache down the line.

Deployment Slots

Azure has something called Deployment Slots. If you’ve read through this article, you’re probably wondering why I haven’t mentioned it earlier! The short version of what they are is the ability to have different “slots” of the same application, and you can swap slots for what’s live. For example, you have a dev slot and a prod slot, and you can automatically swap what’s in the dev slot to prod. You can rollback and have custom “warm up” actions before swapping in.

Sounds pretty neat, right? Well, there’s a reason or two why I didn’t spend this whole time talking about it.

  • It works only for Azure Web Apps
  • You have to have be on a standard or above pricing tier ($$$)
  • You cannot have multiple instances per slot, so you can’t having a testing environment for a multi instance site. That means you cannot test scaling, which is in my mind the entire point of having Azure.

Don’t get me wrong, if you have an all web app system, you should definitely use deployment slots. But, what enterprise system out there is only web app based? Now, could you use a hybrid of this branching system and deployment slots? Totally. But, since what I put together works for any system, I would say you should just use this one.

 

Conclusion

Now you’re all familiar with this system, you can create Kat’s Ice Cream shop to be the best it can be. The production site is up and running, and you can easily duplicate environments. It’s a great system for those that need that. And, now you can go get some ice cream.

 

Update: Just recently found out about Release Management tools that hook into Azure. Looks like a great solution for some people, and might be integral to the system here. At the time of this development we didn’t have access to RM, but it looks neat.