Azure Monitor for application monitoring with Terraform

One of our most recent projects involved the design and deployment of a highly-available (and high security) application platform in Microsoft Azure. In addition, the platform would be deployed several times for several different customers, across several regulatory domains! A repeatable, consistent, and automated deployment process was a key outcome for the project.

A big part of this orchestration is ensuring we have sufficient monitoring of availability and performance across the entire stack including Azure VMs, .NET applications, MS SQL, SSRS, Active Directory and more. Generating alerts based on availability events and configured performance threshold values was a must for the customer.

Azure Monitor Dashboard
Azure Monitor Dashboard

To achieve this we used Terraform, Chef, PowerShell scripts and ARM templates to build Azure Monitor to fit our requirements.

The basic structure for Azure Monitor in this scenario is as follows:

  1. Create Azure storage account for monitoring, Azure Application Insights, Log Analytics Workspace and monitor action group.

  2. Enable Azure Diagnostic monitoring with customised parameters.

  3. Deploy Azure Application Monitor and dependent agent to Azure VMs.

  4. Configure Azure Agents against the Log Analytics Workspace.

  5. Configure Azure live monitoring to capture dotnet application states such as performance, request count, errors.

  6. Configure Azure alerts based on Azure resources and metrics from the Azure VMs.

Building the prerequisites

During the Terraform execution, we build:

  • Azure Storage Account

  • Application Insight instance

  • Log Analytics Workspace, and

  • Monitor Action Group

Here’s a snipped of our Terraform code required to build those resources.

resource "azurerm_storage_account" "test_sa" {
name = "test_sa"
resource_group_name = "${var.rg_name}"
location = "${var.location}"
account_tier = "Standard"
account_replication_type = "LRS"
}

resource "azurerm_application_insights" "test_insight" {
name = "test_insight"
location = "${var.location}"
resource_group_name = "${var.rg_name}"
application_type = "Web"
}

resource "azurerm_log_analytics_workspace" "test_analytics" {
name = "test_analytics"
location = "${var.location}"
resource_group_name = "${var.rg_name}"
sku = "PerGB2018"
retention_in_days = "${var.retention_period}"
}

resource "azurerm_monitor_action_group" "test_ag" {
name = "${var.action_group_name}"
resource_group_name = "${var.rg_name}"
short_name = "action"

webhook_receiver {
name = "${var.webhook_name}"
service_uri = "${var.webhook_url}"
}
}
Azure VM Extension for more metrics

Azure basic monitoring provides limited stats for virtual machines. To enable all the stats, deploying Azure VM extensions for all targeted VMs is required.

We augment our ‘azurerm_virtual_machine’ build module in Terraform with the below, which configures Diagnostic Settings for the Azure VM.

resource "azurerm_virtual_machine_extension" "Diagnosticsettings" {
count = "${var.do_bootstrap == true ? 1 : 0}"
name = "vmext-Diag-${var.vm_hostname}"
location = "${var.location}"
resource_group_name = "${var.rg_name}"
virtual_machine_name = "${azurerm_virtual_machine.windows-vm.name}"
publisher = "Microsoft.Azure.Diagnostics"
type = "IaaSDiagnostics"
type_handler_version = "1.9"

settings = <<SETTINGS
{
"xmlCfg": "${base64encode(var.monitoring-template)}",
"storageAccount": "${var.storage_name}"
}
SETTINGS

protected_settings = <<SETTINGS
{
"storageAccountName": "${var.storage_name}",
"storageAccountKey": "${var.storage_key}"
}
SETTINGS

tags = {
environment = "${var.environment}"
role = "${var.role}"
}
}

Additionally we deploy Azure Application Monitoring and its Dependency Agents.

Using Terraform to install the VM extensions we can bind those agents with Analytics Workspace using the Workspace ID and key values as demonstrated in the snippet below.

resource "azurerm_virtual_machine_extension" "monitor-DependencyAgent-agent" {
count = "${var.do_bootstrap == true ? 1 : 0}"
name = "vmext-monitorDepAgent-${var.vm_hostname}"
location = "${var.location}"
resource_group_name = "${var.rg_name}"
virtual_machine_name = "${azurerm_virtual_machine.windows-vm.name}"
publisher = "Microsoft.Azure.Monitoring.DependencyAgent"
type = "DependencyAgentWindows"
type_handler_version = "9.5"
auto_upgrade_minor_version = true

settings = <<SETTINGS
{
"workspaceId": "${var.analytics_workspace_id}"
}
SETTINGS

protected_settings = <<PROTECTED_SETTINGS
{
"workspaceKey": "${var.analytics_workspace_key}"
}
PROTECTED_SETTINGS

tags = {
environment = "${var.environment}"
role = "${var.role}"
}
}

resource "azurerm_virtual_machine_extension" "monitor-agent" {
count = "${var.do_bootstrap == true ? 1 : 0}"
name = "vmext-monitorAgent-${var.vm_hostname}"
location = "${var.location}"
resource_group_name = "${var.rg_name}"
virtual_machine_name = "${azurerm_virtual_machine.windows-vm.name}"
publisher = "Microsoft.EnterpriseCloud.Monitoring"
type = "MicrosoftMonitoringAgent"
type_handler_version = "1.0"
auto_upgrade_minor_version = true

settings = <<SETTINGS
{
"workspaceId": "${var.analytics_workspace_id}"
}
SETTINGS

protected_settings = <<PROTECTED_SETTINGS
{
"workspaceKey": "${var.analytics_workspace_key}"
}
PROTECTED_SETTINGS

tags = {
environment = "${var.environment}"
role = "${var.role}"
}
}
Recipe Time

Now all the VMs have been created with Azure Monitor in place.

The next step is to configure live monitoring for the dotnet web application our platform will be hosting. As we’re already using Chef to configure the VMs, we create a recipe to run some PowerShell and install the Azure Application Insight agent.

We’re then able to connect the dotnet application to Azure Insight using the Insights ‘InstrumentationKey’.

After configuring all these things, we’re able to see all the application stats from the monitoring dashboard.

powershell_script 'Install Azure Insights' do
code <<-EOH
try {
Set-PSRepository -Name "PSGallery" -InstallationPolicy Trusted
Install-PackageProvider -Name NuGet -Force
Install-Module -Name PowerShellGet -Force
Update-Module -Name PowerShellGet
Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature
Install-Module -Name Az.ApplicationMonitor -AcceptLicense -Force
Import-Module 'C:\\Program Files\\Microsoft Application Insights\\Status Monitor\\PowerShell\\Microsoft.Diagnostics.Agent.StatusMonitor.PowerShell.dll'
Start-ApplicationInsightsMonitoring -Name #{website} -InstrumentationKey #{insight_key}
} catch {
Write-Host $_.Exception.Message
Continue
}
EOH
end
Alerting

Our next goal was to configure alerts for Azure resources, and to do this we return to Terraform. In Terraform we can define the metrics that we need to monitor and what would be the threshold values to trigger alerts.

Not only that, but we can bind those alerts with our alert notification groups as well.

Here’s an example for alerting on Storage IOPS / Transaction thresholds.

resource "azurerm_monitor_metric_alert" "storageAlert01" {
name = "storage-metricalert"
resource_group_name = "${var.rg_name}"
scopes = ["${var.alert_storage_id}"]
description = "Action will be triggered when Transactions count is greater than 50."

criteria {
metric_namespace = "Microsoft.Storage/storageAccounts"
metric_name = "Transactions"
aggregation = "Total"
operator = "GreaterThan"
threshold = "${var.storage_transaction_threshold}"

dimension {
name = "ApiName"
operator = "Include"
values = ["*"]
}
}

action {
action_group_id = "${var.alert_group_id}"
}
}

We had another requirement to configure a few alerts for specific VMs. We had tried to used Terraform to achieve that goal to no avail. So we decided to move with Azure ARM templates and PowerShell scripts to execute all the commands.

In this implementation, we have one common ARM template with all the variables and we have different ARM templates with different parameters for different metrics. Deployment via PowerShell script was simple, using the below commands:

Connect-AzAccount
Select-AzSubscription -SubscriptionName <yourSubscriptionName>
New-AzResourceGroupDeployment -Name MultiResourceAlertDeployment -ResourceGroupName ResourceGroupWhereRuleShouldbeSaved -TemplateFile list-of-vms-static.json -TemplateParameterFile list-of-vms-static.parameters.json
More Information

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on LinkedIn
Share on email
Share through Email