Visual Studio Team Services Release Management for O365 product development.

The scenario.

A team working on a Provider Hosted SharePoint Add-in, hosted on Azure with multiple deployments for different O365 tenants. The development team wants a friction free, controlled and with as less as possible manual tasks delivery of the system to the different customers.

The setup.

The team uses GIT with Visual Studio Team Services to organize their sources, backlog, tasks, builds and releases. A default GitFlow policy is used for the branching structure with a drop to GitHub for the created Open Source components. Azure Resource Manager JSON definitions are used for creating the Azure Resources used by the system.

release2

The Development team

Every team member uses its own developer instance of O365. Actually not all team members, some have Visual Studio MSDN Professional which doesn’t come with a O365 developer tenant. They are working on developer Sites Collections created on the company O365 tenant. See: https://www.visualstudio.com/en-us/products/compare-visual-studio-2015-products-vs.aspx

A normal GitFlow (read: A successful Git branching model) is used by team members with Branch policies on the develop and master branch for Pull Requests, Linked work items and code reviews.

Two developer tips: keep the value of the AppManifest ClientId to a wildcard ‘*’, changing this will break all other developers debugging experience. And the other tip, team members often forget this one, when the SharePoint App hasn’t change, disable it as a startup project when debugging. The often time consuming ‘retracting SharePoint app – installing SharePoint app’ activities are skipped and you can validate the application much faster. You only have to run it with the SharePoint app enabled as startup project once to get the ‘on the fly generated’ ClientId in the app. From than just debug the WebApp only.

VSTS + Zapier + GitHub

Sources are also pushed to GitHub when a commit is done on the master branch via Zapier.

clip_image004

http://clemensreijnen.nl/post/2015/10/29/Sogeti-O365-Site-Provisioning-(on-GitHub)

! Very important, remove all Azure keys from your sources before pushing it to GitHub. Read this post why: https://www.humankode.com/security/how-a-bug-in-visual-studio-2015-exposed-my-source-code-on-github-and-cost-me-6500-in-a-few-hours

Azure Key Vault for Application settings.

To be sure Azure keys, and other application settings, aren’t used for the above mentioned horror scenario the Azure Key Vault is used by the system. The Azure Key Vault is connected to Azure Active Directory and only applications and users registered in the Azure Active Directory are able to access the keys. https://azure.microsoft.com/en-us/services/key-vault/
Happy secure but still the build removes also these keys from the config files, the real keys are next to the web application in the Azure Web App settings pane.

The Build.

3 builds: 2 CI for code, 1 for creating the SharePoint Apps for all environments.

Builds are triggered continuously on develop and master branch commits to compile, validate and package the WebApp and other ‘non-SharePoint’ components. One additional build is responsible for building and making the SharePoint app packages.

clip_image008

The WebApp build has four build steps.

Two are Visual Studio steps, a Visual Studio Build with the build arguments “/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true” for building and packaging the solution, and a Visual Studio Test build step for running the unit tests. So you get the nice looking informative test results.

clip_image010

There is one additional PowerShell build step for the Azure WebJob packaging. For some reason the default package method of Visual Studio MSBuild creates a zip file for the Azure WebJob with an enormous directory hierarchy. This directory hierarchy makes the deployment fail (see release prargraph). The PowerShell build step simply takes the WebJob compile output and puts it in a zip.

clip_image011

The Copy Files build step copies the zip files and the PowerShell files to the drop folder on the server. These are needed for the release process.

See also: http://www.codewrecks.com/blog/index.php/2015/06/30/manage-artifacts-with-tfs-build-vnext/

The Create SharePoint Apps build.

The system needs multiple SharePoint Apps, per customer installment and per stage a SharePoint App. See image, every stages got its own SharePoint app with the same features, but with different security settings.

The deltas per SharePoint App are the ClientId, ClientSecret and the URL to the Web Application (all other deltas are organized via feature flag settings).

Side note: it is easier to make a multitenant solution, register the SharePoint App in the Office Seller dashboard and organize the customer specific settings via feature flags. You only need to register one SharePoint App per stage.

clip_image013

Creating a SharePoint App packages via Visual Studio is easy. Right click on the SharePoint project and select publish. In the dialog select the Publishing profile and move on. For creating multiple SharePoint Apps via VSTS Build the Publishing Profiles are the key items. Create for every needed configuration (ClientId, ClientSecret and the URL to the Web Application) a publish profile. Easy done via Visual Studio.

clip_image015

Via a MSBuild Build step where the SharePoint project is selected and with an MSBuild argument “/p:ActivePublishProfile="ProfileToBuildAppFor" the apps are created and ready to put in the AppCatalog.

clip_image016

Putting the apps in the corresponding AppCatalog’s and update the sites which are using the App is still an annoying manual step. Maybe this can be done with the PNP O365 PowerShell cmdlets, something for investigation.

The Release.

The release process contains Development, Test, Staging and Production environments. Where the development environment is better to be named Development Integration environment, the place where the integration of all developer artifacts are validated.

Resource groups and subscription organization.

The Development and Test environments are put in a different Azure Resource Group than the Staging and Production environments, this to keep a clean separation of costs and access right. Via Azure Active Directory and Resource Group RBAC the development team only has access to the Development and Test environments. Customer system installments live in different Azure subscriptions with Staging and Production in the same subscription and resource group (for swapping).

Release management environments

Per customer a release with environments is configured in VSTS Release management. There is one release with an ‘empty’ environment, this one is used to validate the release of a complete new deployment. Via the release tasks ‘create or update Resource Group’ a complete new resource group of with Azure Resources for the system is created and deleted after the deployment of the bits and the test execution.

clip_image018

This makes the team comfortable to rollout new environments in a repeatable way, and makes sure the Resource Group Template is up to date.

All other releases are ‘upgrade’ releases, the Azure resources are upgraded if necessary. Mainly the upgrade is about changes in the application settings (.config) of the Web App. At this moment the team uses PowerShell to update the settings, should also be possible via the release task ‘Update RG’ but that one is on the investigation backlog.

Develop, Test and Staging have all three release tasks, deploy the Azure WebApp, deploy the Azure WebJob and run Visual Studio Test.

clip_image020 

The WebJob deployment is an Azure PowerShell tasks with the command: “New-AzureWebsiteJob -Name $webSiteName -JobName $jobName -JobType $jobType -Slot $slot -JobFile $packageFile”, it takes the zip file created during the build.

The Develop Branch CI build triggers only the internal release to the develop integration environment and test. While the Master Branch CI build triggers all releases. VSTS Release management at this moment doesn’t have the capability to link or run releases in parallel. To work around this manual approvals are put before releasing to a stage. After the Develop build has gone through the internal release stages, the pre-approval is made for the customer releases to the customer staging environments.

The last release step, from staging to production, is a single PowerShell command “Switch-AzureWebsiteSlot -Name $WebSiteName -Slot1 $SlotNameFrom -Slot2 $SlotNameTo -F” which swaps staging with production.

All environments contain the run test Visual Studio Tests which contains not only the unit tests but also integration tests, validating the communication with the different Azure storage locations and SharePoint O365, making it sure that all settings are made correct. Which is still the most occurred reason why a release fails.

SNAGHTML59f480

Happy comfortable releasing.

Add comment