We are working on a control and monitoring system for on-premises & cloud hosted servers and .NET applications. The monitoring is done by installing a windows service on all the virtual machines that are part of the system. The windows service’s job is to run (and keep open) a console app that collects data from the local machine and sends it to the monitor web service. After deploying the windows service to a dozen machines we realized that updating the collector component involves a lot of time and effort. So we've decided that we need an automated process that will detect when a new version of the collector component has been committed on Git, package the new version and apply it on all the servers.
Our automatic updates system is composed of a TeamCity server and a WebAPI service that collects the data, hosts the update packages and acts as a proxy between the TeamCity server and our windows services. We don’t want direct communication between TeamCity and the windows services since it would scale poorly (we can have multiple WebAPI services to distribute packages when we need to scale) and direct communication between the windows service host and TeamCity might not be wanted/possible.
TeamCity will detect changes in the Git repository of the collector component, pull the changes locally, build the collector project, increment the assembly version if the automated tests have passed, use MsBuild to build the project and archive the artifacts in a zip package. The package, containing the current version number in its name, gets deployed to the WebAPI service that stores it in its local storage.
Getting the update package from TeamCity to the update service can be done in several ways. We can configure TeamCity to deploy the WebAPI service along with the ZIP package via IIS Web deploy, but that would mean that every time a new update shows up the WebAPI service will have to be re-deployed and restarted. Another way to achieve this is to make the WebAPI service check the TeamCity API when the windows service asks it for updates. This is probably the best approach (taking development effort into account), and the downsides can be overcome by making the WebAPI service cache the results for a certain amount of time and only download a new update on the first request. Another approach would be to expose the updates folder as a network drive and XCopy the packages to it, but this requires setting up the TeamCity account to have write access to that folder.
There are several viable options for distributing the update package from the WebAPI service to the various windows services. The WebAPI can either push notifications to the windows service (using SignalR or Redis pub/sub), or the windows services can check for updates with the WebAPI service at some set interval. We’ve chosen to have the windows service poll the WebAPI for updates since it’s easier to implement and it’s not critical for us to update the collector component exactly at the moment a new version is available. If, however, we’ll need to make sure the collector is always up to date, the WebAPI service (which not only acts as an update agent, but also collects all the data sent from the collector) can check the collector’s version on each call it receives, and if a new one is present, tell the collector to signal the windows service that an update is needed.
The window service, once it downloads an update, waits for the collector console to finish the running job and shuts it down, unzips the package and overwrites all the files it contains, then starts the new version of the collector.
Since we’re distributing executables to many mission-critical machines, security is a big issue. Assuming the TeamCity server is secure from external attacks, the problem then lies with the updater service. If a WebAPI update service is compromised, then any machines requesting updates from it will be too. In order to mitigate this, the executables delivered by TeamCity need to be signed with a certificate that enables strong encryption. The windows service will then have to verify that the collector executable is signed before launching it.
We're interested to hear your thoughts on our design, improvements are always welcome.
Posts
0 comments:
Post a Comment