Tag Archives: vsts

Reduce your build time with parallelism in Azure DevOps

Your team works with a project in Azure DevOps. Your build time starts to increase as the project’s complexity grows but you want your CI build to deliver results as quickly as possible. How can you do that? With parallelism, of course!

Let’s do this together.

Prerequisites

Before we start to design a build pipeline with parallelism we must be aware of how Azure DevOps orchestrate parallelism and how many parallel jobs we can start. I recommend to read the official Microsoft Docs page about this.

Designing the build

The following example shows how to design a build with:

  1. A first “initialization” job.
  2. The proper build jobs: build 1 and build 2 that we want to run in parallel after the step 1.
  3. A final step that we want to execute after that build 1 and build 2 are completed.

We start with configuring the build to look like the following picture:

To orchestrate the jobs as we specified before we use the “Dependencies” feature. For the first job we have no dependencies so leave the field blank.

For the Build 1 job we set the value to Init. This way we’re instructing Azure DevOps to start the Build 1 job only after that Init has completed.

We do the same thing with the Build 2 job.

For the final step we set Build 1 and Build 2 as dependencies so this phase will wait for the 2 previous builds to complete before starting.

Here we can see the build pipeline while it’s executing.

TL;DR

With this brief tutorial we learned how to design a build pipeline with dependencies and parallelism that can reduce the delay of our CI processes. A fast and reliable CI process is always a good practice because we must strive to gather feedback as quickly as possible from our processes and tools. This way we can resolve issues in the early stages of our ALM, keeping the costs down and avoiding problems with customers.

How to release a hotfix with pull-request inside VSTS in 3 steps

We all know the situation: the customer finds a critical bug in the latest release and he wants us to release a new version of our application with a fix. How do we handle this situation without breaking our team policies? How to release a specific fix to avoid regression problems?

First we need to fix that bug with the classical approach of feature-branches. We create a branch, fix the bug, create a pull-request and the team approves. These activities are set at the highest priority because a customer is in trouble and we must help.

Now we are in a situation where we have a specific commit that solves a specific problem with only the necessary lines of code modified.

commit-with-fix.png

Our target is to ship that specific commit that fixes that specific bug with a standard pull-request, without all the other work done in the development branch since the latest release. So we write down (in our clipboard for example) the id of the commit.

shaid

What can we do now to release?

1. New branch

We create a new branch.

git checkout -b my-hotifx

Now we fetch the latest updates from the remote repo.

git fetch origin

Then we set our branch my-hotfix to point to the latest commit of the remote release branch.

git reset --hard origin/release

branch-release.png

2. Cherry-pick

Now we cherry-pick the specific commit we want to apply to the release branch without commiting (–no-commit option). We choose the no-commit option to carefully inspect what is going on in our files. It’s here where we use the commit id we saved early.

git cherry-pick <commit-hash> --no-commit

We verify that everything is fine (where fine depends on your specific project). Now we can commit and push to VSTS.

git add .
git commit -m "Hotfix"
git push origin mybranch:mybranch

final-situation-1

3. Open pull-request

Our branch is now on the remote repo and we can open the pull-request with VSTS.

pull-request.png

From here the team can approve the PR and fire up our automated release process that activates our automated test and if everything is fine we deploy safely.

TL; DR

With this blog post we explored the critical situation to release a hotfix without breaking the rules or taking shortcuts to avoid our release pipeline. As engineers we must maintain a cold approach even in hot situation and rely on our best practices. Human errors are always possible in particular when we’re stressed and a customer is making our phones hot.

Mastering database migrations scripts with VSTS

During the lifecycle of our applications it’s very likely that to support new functionalities we need to modify the structure of our database. In this particular post I mean a relational database.

When we deal with a database in a production environment we can’t simply drop an existing database and create a new one with the latest version of the schema.

The database is unlike application code in as much as it contains state, in terms of table-based data,  that needs to be preserved after an upgrade.

drop-create

This is why when dealing with database changes we need to talk about migration scripts.

Migration Scripts

A migration script is a set of SQL commands that changes the database schema or updates data to evolve the database from the version n-1 to the version whitout loss of data.

Migration scripts are typically collected in a dedicated folder in our repository, named sequentially.

The migration scripts must be executed in the specific order to be able to build the n-th version of our database.

The first migration script

The first script in our collection is responsible to create the first version of our db starting from an empty database.

The following is an example of a simple first migration script that we can name 00000001.sql. The example is for a simple database with one schema named Ordering and one table called Customers in the Ordering schema.

CREATE SCHEMA Ordering
GO
CREATE TABLE Ordering.Customers(
    Id INT PRIMARY KEY,
    CompanyName NVARCHAR(100) NOT NULL
)
GO

Create the first migration script from an existing database

If we work on a solution with an existing complex database we need a tool to create our first migration script. Tools are capable of tracking dependencies and relations inside the db schema to calculate a single script that evolves our db to the desired state. There are many tools on the market. In this example we’ll use Red-Gate SQL Compare. With this tool we select our existing db as the source (IC6.ParrishShoes) and our target to sync. As we explained before we need an empty database so we compare our db to that and we obtain our first script.

The script calculated by SQL Compare engine is the following.

/*
Run this script on:

        ..EmptyTarget    -  This database will be modified

to synchronize it with:

        ..IC6.ParrishShoes

You are recommended to back up your database before running this script

Script created by SQL Compare version 13.4.0.6563 from Red Gate Software Ltd at 03/08/2018 23:20:53

*/
SET NUMERIC_ROUNDABORT OFF
GO
SET ANSI_PADDING, ANSI_WARNINGS, CONCAT_NULL_YIELDS_NULL, ARITHABORT, QUOTED_IDENTIFIER, ANSI_NULLS ON
GO
SET XACT_ABORT ON
GO
SET TRANSACTION ISOLATION LEVEL Serializable
GO
BEGIN TRANSACTION
GO
IF @@ERROR  0 SET NOEXEC ON
GO
PRINT N'Creating schemas'
GO
CREATE SCHEMA [Ordering]
AUTHORIZATION [dbo]
GO
IF @@ERROR  0 SET NOEXEC ON
GO
PRINT N'Creating [Ordering].[Customers]'
GO
CREATE TABLE [Ordering].[Customers]
(
[Id] [int] NOT NULL,
[CompanyName] [nvarchar] (100) COLLATE Latin1_General_CI_AS NOT NULL
)
GO
IF @@ERROR  0 SET NOEXEC ON
GO
PRINT N'Creating primary key [PK__Customer__3214EC072CF93715] on [Ordering].[Customers]'
GO
ALTER TABLE [Ordering].[Customers] ADD CONSTRAINT [PK__Customer__3214EC072CF93715] PRIMARY KEY CLUSTERED  ([Id])
GO
IF @@ERROR  0 SET NOEXEC ON
GO
COMMIT TRANSACTION
GO
IF @@ERROR  0 SET NOEXEC ON
GO
-- This statement writes to the SQL Server Log so SQL Monitor can show this deployment.
IF HAS_PERMS_BY_NAME(N'sys.xp_logevent', N'OBJECT', N'EXECUTE') = 1
BEGIN
    DECLARE @databaseName AS nvarchar(2048), @eventMessage AS nvarchar(2048)
    SET @databaseName = REPLACE(REPLACE(DB_NAME(), N'\', N'\\'), N'"', N'\"')
    SET @eventMessage = N'Redgate SQL Compare: { "deployment": { "description": "Redgate SQL Compare deployed to ' + @databaseName + N'", "database": "' + @databaseName + N'" }}'
    EXECUTE sys.xp_logevent 55000, @eventMessage
END
GO
DECLARE @Success AS BIT
SET @Success = 1
SET NOEXEC OFF
IF (@Success = 1) PRINT 'The database update succeeded'
ELSE BEGIN
	IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION
	PRINT 'The database update failed'
END
GO

Compile a database

If we run the previous script and we get zero errors we can say that we can compile our database. The same concept apply when we can execute a collection of migration scripts in the correct order without errors.

Being able to compile a database it’s a huge benefit because it enables us to create a database on demand by simply executing a list of script to get the desired version on which we can develop, run test, deploy to a new customer etc. We can also start to do some automation for our database tasks.

Automation with VSTS – Checking our migration scripts

An important part of having a collection of db migrations scripts is that we have to frequently check that this chain of scripts is valid.

We can invest in automation to do that and with VSTS is very simple to get started.

In the following example we create a build process that executes all the migrations scripts in our repository to a target database to check if everything can be completed without errors.

We go to the Build section of VSTS and create a new Build process from an empty process.

We browse in the marketplace the DbUp extension (free!).

DbUp is a .NET library that helps you to deploy changes to SQL Server databases. It tracks which SQL scripts have been run already, and runs the change scripts that are needed to get your database up to date.

2018-08-03_23-40-20

After the installation process of DbUb we can refresh our list of available task and add the DbUp Migration task. With this task we can execute scripts included in a folder of our repository to a target database. The configuration of the connection string is done with VSTS Build variables in the dedicated tab. We configure the script folder path and the Single Transaction strategy, too. This way all the scripts are executed in one single transaction and if we get an error everything will be rolled-back.

2018-08-04_00-06-31

2018-08-04_00-16-49.png

When we complete the configuration of the task we click Save And Queue.

The build process will start and if we’re a bit lucky everything will succeed.

2018-08-04_00-24-07.png

We can see that DbUp applied correctly the scripts by browing the db with SQL Mangament.

  • First 2018-08-04_00-07-34
  • After 2018-08-04_00-12-42

TL; DR

With this tutorial we’ve learned the basics of migrations scripts, how to create them with a tool and how to check if our collection of scripts is valid with automation implemented with VSTS and DbUp.

This is just the start

When we have such powerful tools we can start to think about automated processes that test our chain of scripts in many ways. We can do an “incremental” approach like the example above triggered at every change in the scripts folder. We can schedule a build process that every day at noon will execute all the migration scripts versus a temporary brand new database.

What will you do with this super powers? Let me know in the comments below!

Reference

VSTS DbUp Marketplace – https://marketplace.visualstudio.com/items?itemName=johanclasson.UpdateDatabaseWithDbUp

Red-Gate SQL Compare – https://www.red-gate.com/products/sql-development/sql-compare/index

Power is nothing without control

You can’t control (leave alone improve) what you can’t see – Me

Power is nothing without control – Pirelli

Why a dashboard?

When we drive our car we have everything under control: speed, revs, oil and water temperatures and fuel level. We need a dashboard in our car because we have to know our current speed to respect speed limits, to know how much petrol we have in our fuel-tank to decide if we can go to work without a trip to the gas station etc. All this data to do the simple job of driving! This is necessary because we need to take informed decisions.

white motorcycle cluster gauge

Photo by Mikes Photos on Pexels.com

What does it mean for our daily activities in a software department? We need our dashboard, too! How can we do our job of deliveing (not only writing!) a working software with the lowest bug count as possibile, quickly, on a budget and coordinating with other people? It’s a huge task compared to drive a car alone and yet many of us rely on instinct and guts to make decisions for an entire software team.

A dashboard for the software Engineer

Which indicators and gauges do we need as software engineers? I think there are a few things we Always need to know about our team.

  1. Team Cycle time: how much time/days does a unit of work take to go from started to completed? And with completed we mean delivered to the customer.
  2. Team Lead time: how much time/days dows a unit of work take to go from a created to completed?
  3. How fast are we going? How many story points we deliver every fixed amount of time? (sprint, week… name your favorite).
  4. Bug count. How many known defects (bugs) have we? Is the count increasing or decreasing?

Here we can see a dashboard created with Microsoft VSTS where the team can get an immediate report of the situation.

2018-08-03_13-46-49.png

Create a dashboard

Every team is different and needs specific/custom dashboard. To create a dashboard with Microsoft VSTS we can go to

https://THE_DOMAIN/THE_PROJECT/_dashboards/

and then we expand the dashboard list with the arrow button (1) and click New Dashboard (2).

2018-08-03_14-02-31.pngWe give our dashboard a name and hit Create.

2018-08-03_14-09-48.pngWe can add widgets picking from the right-side list, search or browse the Marketplace to add something extra to our VSTS.

2018-08-03_14-11-55.png

When we’ve finished to create our dashboard we click “Done editing”.

TL; DR

A dashboard is a must have to control or improve our team and our process. It enables us to understand our current indicators and shows if a new way to work improves or worsen the situation. With a dashboard we can take informed decisions about many arugments: current velocity of the team, quality of the software, lead time and so on. We’ve seen how to create and configure a dashboard with Microsoft VSTS.

Reference

Microsoft VSTS Docs: https://docs.microsoft.com/en-us/vsts/report/dashboards/?view=vsts

Import from JIRA to VSTS in 5 steps

In previous posts I documented some features of VSTS and a few scenarios of Application Lifecycle Management.

If you like VSTS it’s likely that you want to move your existing project from your current platform to VSTS. I’m currently planning a migration from JIRA to VSTS at work and here I am to share my findings.

This blog post has been inspired by the work of the ALM DevOps Rangers.

We’re going to build a simple tool to migrate and map Jira issues into VSTS items.

The source code is available on GitHub. Continue reading

VSTS for beginners: release your web-app to Azure

In the previous post of this series dedicated to VSTS we talked about continuous integration. Now we’ll talk about publishing our Web-App hosted on Azure with the release management tools provided by VSTS. With this kind of tools we can deploy the latest version of our web-app to Azure in complete automation removing manual procedures and human errors. The setup of Azure will be covered in another blog post.

 

Continue reading

VSTS for beginners: improve quality with continuous integration in 3 easy steps

In this blog post we’re going to configure a build process in VSTS to enable continuous integration for our ASP.Net Core example web-app.
Continuous integration is a powerful technique to prevent merge-hell and improve quality on the “left” stages of our software production process. In the fast-paced world of development we want to merge into the main line of development the new developed features as soon as possibile to avoid open branches that will cause painful merges. If we keep our unit of work small and focused we’ll have great benefits.

Continue reading

Improve leadtime and workflow with WIP limits on VSTS

A powerful method to keep or make a customer happy is to satisfy his needs quickly. In our typical very busy day we do a lot of things and this hurts our ability to concentrate and focus. This is true for us individually but this is also true as a team.

Little’s Law

We can improve the response time to our customer by leveraging the Little’s Law:

It is a theorem by John Little which states that the long-term average number L of customers in a stationary system is equal to the long-term average effective arrival rate λ multiplied by the average time W that a customer spends in the system. – Wikipedia

L = λ W

For example if our team completes 2 user stories every day (𝜆 = 2) and every user story takes 4 days to be developed (W = 4) we have a system with L = 8 ongoing activities.

If we write Little’s Law like this

W = L / λ

we discover that wait time (W) can be reduced if we reduce L, the amount of ongoing activities at the same time.

VSTS boards and WIP-limits

If you manage your team activities with Visual Studio Team Services (VSTS) you can improve the flow and reduce lead-time by limiting the amount of work in progress in your kanban board.

You can setup the WIP-limits in the board of your Backlogs. Go to Work -> Backlogs and then click on Board.

2.png

As you can see VSTS has the default value of 5 for the Active and Resolved statuses. It means that your team cannot drag more than 5 PBIs in the Active or Resolved column otherwise the number of activities turns red, indicating that your team is doing too much work at the same time.

1.png

Default settings of VSTS

3.png

Too much activities!

You can change the values for to better fit the needs and size of your team by clicking the gear icon in the board and access the settings and then click save.
Of course you can also change the column names and add other statuses as well but we’ll cover this in another blog post.

4.png

Where do I start?

A very low WIP-limit can freeze the activities of your team and it may be very hard to respect. Don’t spend lots of time to calculate or debate about WIP limit: start with number that gives flexibility, plan a meetings two week later and discuss it again. You and your team will find the right balance.

TL;DR

Limiting the work in progress is a powerful way to improve the lead-time of our activities and it’s very cheap, too! It’s a team discipline activity and at the beginning it can be counter-intuitive and hard to respect. When a team is doing too much activities at the same time the quality goes down and the lead-time goes up and rework will be required. If the WIP limits block you from starting a new user story you can pair programming with a colleague or reply to that old e-mail that’s waiting so long.

Picture1.gif

 

Agile@School – episodi otto e nove

Si è concluso oggi il progetto Agile@School presso l’IIS di Rovigo che ha coinvolto alcuni studenti delle classi quinte indirizzo informatica.

pexels-photo-267885.jpeg

Il post di oggi riassume gli ultimi due incontri (l’ottavo e il nono) dove abbiamo portato a termine un’app con Xamarin Forms per accedere a Twitter come output finale.  Continue reading

Agile@School – settima lezione

Settimo appuntamento presso la scuola rodigina IIS Viola/Marchesini per il progetto Agile@School nella giornata di ieri 24 gennaio 2018. Si è giunti alla parte finale dove cerchiamo di mettere insieme tutte le parti finora studiate. Ci siamo dati come obiettivo quello di realizzare una semplice app per l’accesso a Twitter mettendo in gioco le conoscenze apprese.

1.png Continue reading