The Microsoft Visual Studio Setup Project is an old technology to create installer developed by Microsoft. It is out of support from nearly a decade and not present in Visual Studio anymore but when I visit customer sites I find legacy technologies and I need to deal with it on the short-term.
A couple of days ago I was working on an automated CI build on Azure DevOps and we hit an issue when trying to compile an old VDPROJ (migration to Wix in progress, btw ☺). We encountered an HRESULT 8000000A error.
How to fix
After some googling (with Bing) we discovered that it’s a known issue but as the technology is out of support the only that you can do is to apply a workaround.
You need to install the Microsoft Visual Studio Installer Projects extension for Visual Studio on your build machine.
This is because it includes a simple tool as you can read at the bottom of the page:
So what you need to do in your CI process (or Azure Pipeline) is to call this DisableOutOfProcBuild.exe from inside the folder of the exe itself and then build your VDPROJ.
Under the hood
What the simple executable does is changing a value on the registry to enable a deprecated feature of Visual Studio (out of proc build).