YAML pipelines

by Jesper Mayntzhusen, posted on

Heads Up!

This article is several years old now, and much has happened since then, so please keep that in mind while reading it.

NOTE: If you are completely unfamiliar with Azure DevOps pipelines then this article may be too advanced - I'd recommend checking out Aaron Sadlers excellent 3 part series on Skrift.io!
Part 1 - Part 2 - Part 3

Setting up a YAML build pipeline

First thing to do is to add a .yml file to your repo.
In my repo I will add a pipelines folder where I will store all my pipeline things - but you can place the files wherever you want. First off I will add a build.yml file which should handle building the artifact - later we will add a deploy.yml file as well which will take the build artifact and deploy it to a server.
pr: none # triggers on PRs by default, have to opt out
trigger:
  branches:
    include:
    - dev
    - main
  paths:
    include:
    - src
  batch: True
name: Build-$(date:yyyyMMdd)$(rev:.r)

stages: 
  - stage: build
    jobs:
    - job: build
      displayName: Build and save as artifact

      pool:
        vmImage: windows-2019

      steps:
      - checkout: self
So this initial file sets some of the configuration for the pipeline:
  • We don't want it to trigger on PRs (if you need that you can just remove the line entirely as that is the default)
  • We only want it to trigger on specific branches - in my case dev & main
  • We only want it to trigger when files from the src folder are changed - so no runs on e.g. gitignore or readme changes. This can be either an include or exclude list, and will likely need to be amended as things are added.
  • We want it to batch commits - this means if the build is running and 3 new commits are made while it runs, we don't let it run 3 more times. Instead we batch all 3 commits into one run.
  • We set the name to be the date + run number, so we will get names like #Build-20211112.3
Next we set up our first stage - which is just a group of jobs run together - and determine the build agent and the first step which checks out our site's git repository.

At this point the build pipeline doesn't actually build anything, but the basics are there for us to start working on which steps we need to perform to get our build artifact.

Determining build steps

If you think about how you start a site up locally from a repo it is probably something like this:
  • Clone the repo
  • Open the solution
  • Build the site in Visual Studio
  • Navigate to your frontend folder and run npm install
  • While in the FE folder, run npm run build
  • The site now runs
The YAML steps are basically the same, except we also need to ensure that things are installed. So our .yml file already has the checkout step - let's add these steps:
  • Open the solution (Not really possible on a build VM)
  • Build the site in Visual Studio
      steps:
      - checkout: self

      # Install NuGet to restore packages
      - task: NuGetToolInstaller@1
        displayName: 'Use NuGet '

      # Restore packages based on the solution file
      - task: NuGetCommand@2
        displayName: NuGet restore
        inputs:
          solution: v8-testsite.sln

      # Build the solution using MSBuild
      - task: VSBuild@1
        displayName: Build solution v8-testsite.sln
        inputs:
          solution: v8-testsite.sln
          vsVersion: "16.0"
          msbuildArgs: /p:DeployOnBuild=true 
            /p:WebPublishMethod=Package 
            /p:PackageAsSingleFile=true 
            /p:SkipInvalidConfigurations=true 
            /p:PackageLocation="$(build.artifactstagingdirectory)\\" 
            /p:IncludeSetAclProviderOnDestination=False
          platform: any cpu
          configuration: release
So to open and build the solution we need to add these 3 new tasks - first we need to install NuGet before we can use it. Then we restore all packages - this happens automatically in Visual Studio, but has to be done specifically in the pipeline - then finally we use VSBuild which is a command line equivalent to building in Visual Studio. I've added some default config that we use, you can look them up if you are curious what they each do.

Next we want to do the FE (frontend) steps:
  • Navigate to your frontend folder and run npm install
  • While in the FE folder, run npm run build
While the backend build is fairly straightforward since it just runs based on a solution file you specify - frontend can work in many different ways. In my example here I will assume that there is a frontend folder in the root of the repository where we can run the npm commands:
      - task: NodeTool@0
        inputs:
          versionSpec: '12.x'
          displayName: 'Install Node.js'

      - powershell: cd frontend; npm ci
        displayName: 'Install npm dependencies'

      - powershell: cd frontend; npm run build
        displayName: 'Build Frontend'
In PowerShell you can run multiple commands at once by separating them with a ;. Since all tasks default to the root of the site which is where the cloned repo go, we can chain commands as shown above to ensure we run from the correct folder.

Note: npm ci is a more performant version of npm i - read more here.

So at this point the pipeline file should do all the things we need for the build - final part for us is to save the build artifacts so we can deploy them.

Saving the backend files as an artifact are quite easy as the VSBuild task automatically zips it up and places it wherever you specify in the /p:PackageLocation parameter.

So we add the publish artifact task at the end:
      # Save the build output as an artifact to use in the deploy pipeline
      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          ArtifactName: 'BE'
          publishLocation: 'Container'
Our frontend build places all FE files in a specific folder and we zip that up and save it as an artifact, but since the FE builds can be so different it may require other steps for you. The important part is to save all the files you need to place on the server - here is an example of how it can be done:
      - task: ArchiveFiles@2
        displayName: Archive frontend/dist
        inputs:
          rootFolderOrFile: frontend/dist
          archiveFile: $(Build.ArtifactStagingDirectory)/FE.zip

      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact: FE'
        inputs:
          PathtoPublish: $(Build.ArtifactStagingDirectory)/FE.zip
          ArtifactName: FE
At this point our file looks something like this:
pr: none # triggers on PRs by default, have to opt out
trigger:
  branches:
    include:
    - dev
    - main
  paths:
    include:
    - src
    - frontend
  batch: True
name: Build-$(date:yyyyMMdd)$(rev:.r)

stages: 
  - stage: build
    jobs:
    - job: build
      displayName: Build and save as artifact

      pool:
        vmImage: windows-2019

      steps:
      - checkout: self
      
      # Install NuGet to restore packages
      - task: NuGetToolInstaller@1
        displayName: 'Use NuGet '

      # Restore packages based on the solution file
      - task: NuGetCommand@2
        displayName: NuGet restore
        inputs:
          solution: v8-testsite.sln

      # Build the solution using MSBuild
      - task: VSBuild@1
        displayName: Build solution v8-testsite.sln
        inputs:
          solution: v8-testsite.sln
          vsVersion: "16.0"
          msbuildArgs: /p:DeployOnBuild=true 
            /p:WebPublishMethod=Package 
            /p:PackageAsSingleFile=true 
            /p:SkipInvalidConfigurations=true 
            /p:PackageLocation="$(build.artifactstagingdirectory)\\" 
            /p:IncludeSetAclProviderOnDestination=False
          platform: any cpu
          configuration: release

      # Save the build output as an artifact to use in the deploy pipeline
      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          ArtifactName: 'BE'
          publishLocation: 'Container' 

      # Install node
      - task: NodeTool@0
        inputs:
          versionSpec: '12.x'
          displayName: 'Install Node.js'

      # Restore node packages
      - powershell: cd frontend; npm ci
        displayName: 'Install npm dependencies'

      # Run frontend build
      - powershell: cd frontend; npm run build
        displayName: 'Build Frontend'

      # Zip FE files
      - task: ArchiveFiles@2
        displayName: Archive frontend/dist
        inputs:
          rootFolderOrFile: frontend/dist
          archiveFile: $(Build.ArtifactStagingDirectory)/FE.zip

      # Save the FE files as artifact
      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact: FE'
        inputs:
          PathtoPublish: $(Build.ArtifactStagingDirectory)/FE.zip
          ArtifactName: FE
Assuming all paths and required files exist this should work just fine. However both NPM and NuGet packages would currently be downloaded every time, so we could add some cache tasks which would then check if e.g. package.config has changed, and if not it gets the packages from cache instead.

There are some predefined jobs that handles all this for you, the docs have a nice list of common cases here.

Creating the build pipeline in Azure DevOps

First of all - make sure you push your new pipeline file to your repo before we start!

Next, head to Azure DevOps, create a new project and go to the Pipelines section where we can start setting up.

Click the Create Pipeline button, select where your repo is stored, in my case it will be Github YAML.
Once you have found and selected your repo make sure to select Existing Azure Pipelines YAML file! There you can find your YAML file and select it

Select pipeline
It will ask you to review the YAML, and finally you can either Run it or just save it. I recommend saving it then pushing a small change to a file within the included paths to see if it triggers when it should.

It gets a default name which isn't that nice, but if you go back to the Pipelines overview you can rename it to something more fitting - I will call mine build:

Rename pipeline

Setting up a YAML deploy pipeline

If you are familiar with Azure DevOps for build pipelines using the classic editor then it should all feel very familiar with the YAML steps - it is actually super quick to change to since all jobs have a YAML definition you can see in the classic editor and copy over. However, when deploying there are a few more differences - one of the big ones is deployments won't be under the Releases section, but just another pipeline.

NOTE: The example I will use here is for deploying to a VM - there should be jobs for Azure Web Apps and for containers as well, but it's not something I have worked with.

Let's try to set it up - first of all in the pipelines folder I will add a deploy.yml file:
trigger: none # have to set it to NOT be triggered "globally" and then the real trigger is under the pipeline resources
pr: none # triggers on PRs by default, have to opt out

resources:
  pipelines:
    - pipeline: build-pipeline # this is sort of an alias that can be used later
      source: 'build' # the name of the build pipeline in DevOps
      trigger: true # will trigger once the build on this pipeline is done

stages: 
  - stage: deploy
    jobs:
    - deployment: DeployWeb
      displayName: deploy to vm
      pool:
        vmImage: windows-2019
      environment: 
        name: # will set this later
        resourceType: VirtualMachine
      strategy:
        runOnce:
          deploy:
            steps:
A few things are different here compared to the build - here we will ignore the "regular" triggers like commits and prs to the repo, and instead we include a resource - which is our build pipeline from earlier. That means we will only trigger this deploy pipeline when the build pipeline finishes.

We then add a deploy stage which has an environment specified - if you've worked with Deployment Groups in release pipelines before then these environments are very similar. The name is left blank for now as we need to set it up in DevOps later.

For the actual steps we need, we will use the IIS manage and IIS Deploy tasks:
            # Set up the IIS profile we want to deploy to, including hostname bindings
            - task: IISWebAppManagementOnMachineGroup@0
              inputs:
                IISDeploymentType: 'IISWebsite'
                ActionIISWebsite: 'CreateOrUpdateWebsite'
                WebsiteName: 'testsite'
                WebsitePhysicalPath: F:\Websites\testsite
                WebsitePhysicalPathAuth: 'WebsiteUserPassThrough'
                AddBinding: true
                Bindings: '{"bindings":[{"protocol":"http","ipAddress":"All Unassigned","port":"80","hostname":"testsite.domain","sslThumbprint":"","sniFlag":false}]}'
                CreateOrUpdateAppPoolForWebsite: true
                AppPoolNameForWebsite: 'testsite'
                DotNetVersionForWebsite: 'v4.0'
                PipeLineModeForWebsite: 'Integrated'
                AppPoolIdentityForWebsite: 'ApplicationPoolIdentity'
                ParentWebsiteNameForVD: 'testsite'
                VirtualPathForVD: #leave this empty or it breaks
                ParentWebsiteNameForApplication: 'testsite'
                VirtualPathForApplication: #leave this empty or it breaks
                AppPoolNameForApplication: #leave this empty or it breaks
                AppPoolName: 'testsite' 

            # Deploy the BE to the site
            - task: IISWebAppDeploymentOnMachineGroup@0
              inputs:
                WebSiteName: 'testsite'
                Package: '$(Pipeline.Workspace)\build-pipeline\BE\web.zip'
                RemoveAdditionalFilesFlag: false
                ExcludeFilesFromAppDataFlag: true
                TakeAppOfflineFlag: true

            # Deploy the FE to the site
            - task: IISWebAppDeploymentOnMachineGroup@0
              inputs:
                WebSiteName: 'testsite'
                Package: '$(Pipeline.Workspace)\build-pipeline\FE.zip'
                RemoveAdditionalFilesFlag: false
                ExcludeFilesFromAppDataFlag: true
                TakeAppOfflineFlag: true
There are a lot of config options on these, if in doubt look them up in the documentation - but what we basically do here is:
  • Create or update an IIS website called testsite
  • Set that website's path to F:\Websites\testsite
  • Set its domain to testsite.domain
  • Create or update the app pool for it called testsite
  • Deploy our BE artifact to the testsite IIS profile
  • Deploy our FE artifact to the testsite IIS profile

Setting up a deploy environment

The final piece missing for the deploy YAML file is to set the deploy target VM, which needs to be configured in DevOps first.

So if you go to DevOps and click on the Environments tab, then Create Environment. Then fill out a name, I will call mine Dev server, and select Virtual Machine. Finally you get a PowerShell script you can copy.
Add environment script
Now you want to open a remote connection to your server, open PowerShell in admin mode and paste the script in. Run through the wizard (I normally go with all default settings), and finally you should see it succeed, and can now see the environment on the environments tab in DevOps

Check environment

Only thing left to do is to add the name to the YAML file - in my case the name is Dev server. The final file will look something like this:
trigger: none # have to set it to NOT be triggered "globally" and then the real trigger is under the pipeline resources
pr: none # triggers on PRs by default, have to opt out

resources:
  pipelines:
    - pipeline: build-pipeline # this is sort of an alias that can be used later
      source: 'build' # the name of the build pipeline in DevOps
      trigger: true # will trigger once the build on this pipeline is done

stages: 
  - stage: deploy
    jobs:
    - deployment: DeployWeb
      displayName: deploy to vm
      pool:
        vmImage: windows-2019
      environment: 
        name: Dev server
        resourceType: VirtualMachine
      strategy:
        runOnce:
          deploy:
            steps:
            # Set up the IIS profile we want to deploy to, including hostname bindings
            - task: IISWebAppManagementOnMachineGroup@0
              inputs:
                IISDeploymentType: 'IISWebsite'
                ActionIISWebsite: 'CreateOrUpdateWebsite'
                WebsiteName: 'testsite'
                WebsitePhysicalPath: F:\Websites\testsite
                WebsitePhysicalPathAuth: 'WebsiteUserPassThrough'
                AddBinding: true
                Bindings: '{"bindings":[{"protocol":"http","ipAddress":"All Unassigned","port":"80","hostname":"testsite.domain","sslThumbprint":"","sniFlag":false}]}'
                CreateOrUpdateAppPoolForWebsite: true
                AppPoolNameForWebsite: 'testsite'
                DotNetVersionForWebsite: 'v4.0'
                PipeLineModeForWebsite: 'Integrated'
                AppPoolIdentityForWebsite: 'ApplicationPoolIdentity'
                ParentWebsiteNameForVD: 'testsite'
                VirtualPathForVD: #leave this empty or it breaks
                ParentWebsiteNameForApplication: 'testsite'
                VirtualPathForApplication: #leave this empty or it breaks
                AppPoolNameForApplication: #leave this empty or it breaks
                AppPoolName: 'testsite' 

            # Deploy the BE to the site
            - task: IISWebAppDeploymentOnMachineGroup@0
              inputs:
                WebSiteName: 'testsite'
                Package: '$(Pipeline.Workspace)\build-pipeline\BE\web.zip'
                RemoveAdditionalFilesFlag: false
                ExcludeFilesFromAppDataFlag: true
                TakeAppOfflineFlag: true

            # Deploy the FE to the site
            - task: IISWebAppDeploymentOnMachineGroup@0
              inputs:
                WebSiteName: 'testsite'
                Package: '$(Pipeline.Workspace)\build-pipeline\FE.zip'
                RemoveAdditionalFilesFlag: false
                ExcludeFilesFromAppDataFlag: true
                TakeAppOfflineFlag: true
You may be used to having an approval process for release pipelines in the non YAML version - for YAML it is not something you set per stage, instead you can set it per environment. You do so by going to the environment in DevOps, click on it then find the options here:

Set approval

Final thing to do, is to add a new pipeline for deploying, it is the exact same steps as above:
  • Make sure your YML file is pushed to your repo
  • Go to the pipelines section
  • Click the New Pipeline button
  • Select where your repo is stored, in my case it will be Github YAML
  • Once you have found and selected your repo make sure to select Existing Azure Pipelines YAML file!
  • Find your deploy.yml file and select it
  • Rename it to something better - mine will be deploy
At this point when you see it in your pipelines overview you can push another change to trigger the build, and then see if the deploy gets triggered after.

We have now successfully set up YAML pipelines for our site, and it will automatically build and deploy when you commit to one of the tracked branches.

I hope this article has been helpful. Feel free to write me on twitter @JesperMayn with any comments, questions or feedback 😊

Jesper Mayntzhusen

Jesper is on Twitter as