Skip to main content
Version: 1.2.0

Deployment: On Azure App Service

This guide explains how to deploy DataStream on Azure App Service, Microsoft's fully managed platform for building, deploying, and scaling web applications. This deployment model offers simplified operations, automatic scaling, and reduced maintenance overhead.

Benefits​

Deploying DataStream on Azure App Service provides several advantages:

  • Fully managed infrastructure: No VM management or OS patching required
  • Automatic scaling: Built-in auto-scaling based on load or schedule
  • High availability: Guaranteed uptime with SLA backing
  • Easy deployment: Multiple deployment options including CI/CD integration
  • Cost efficiency: Pay only for the resources you use

Limitations to Consider​

Before choosing App Service deployment, be aware of these limitations:

  • Network ports: Limited control over networking (TCP/UDP listeners)
  • Resource constraints: Maximum resource limits per App Service plan
  • File system access: Restricted file system permissions in some cases
  • Protocol support: Limited support for non-HTTP/HTTPS protocols

App Service Plan Selection​

Choose the appropriate App Service plan based on your workload:

PlanCPU(cores)Memory (GB)Recommended For
B111.75Development and testing
P1V213.5Light production workloads
P2V227Medium production workloads
P3V2414High-volume processing

For production environments, Premium V2 or Premium V3 plans are recommended to ensure consistent performance.

Deployment Steps​

1. Create Azure App Service​

  1. Navigate to the Azure Portal and create a new Web App -

    • Select .NET 6 as the runtime stack
    • Choose Linux as the operating system
    • Select or create a new App Service Plan
    • Enable Application Insights for monitoring

  2. Configure deployment options - Select your preferred deployment method (GitHub Actions, Azure DevOps, or manual deployment)

  3. Configure networking -

    • Enable VNET integration if you need to connect to private networks
    • Configure inbound traffic settings in the Networking section

2. Configure App Settings​

Set up the required environment variables in the App Service Configuration:

  1. Navigate to Configuration > Application settings and add the following:

    ParameterSetting
    DATASTREAM_CONFIG_MODEazure_blob (to load config from blob storage)
    DATASTREAM_STORAGE_CONNECTIONYour Azure Storage connection string
    DATASTREAM_CONFIG_CONTAINERBlob container name (e.g. datastream-config)
    DATASTREAM_CONFIG_PATHPath to config files (e.g. Director/config/)

    Set also any additional secret values referenced in your configuration

  2. Configure general settings -

    • Set Always On to On to prevent the app from being unloaded
    • Configure appropriate platform settings (32-bit vs 64-bit)

3. Prepare Configuration Storage​

  1. Create an Azure Storage account if you don't have one -

    az storage account create `
      --name datastreamconfig `
      --resource-group your-resource-group `
      --location eastus `
      --sku Standard_LRS

  2. Create a blob container -

    az storage container create `
      --name datastream-config `
      --account-name datastreamconfig

  3. Create and upload configuration files -

    • Create config directory

      New-Item -ItemType Directory -Force -Path config

    • Create main config file

      @"
      devices:
        - id: 1
          name: http_collector
          type: http
          properties:
            port: 8080
            url: "/api/logs"

      pipelines:
        - name: basic_processing
          processors:
            - grok:
                field: message
                patterns:
                  - "%{COMMONAPACHELOG}"

      targets:
        - name: azure_storage
          type: azure_blob
          connection_string: "${DATASTREAM_STORAGE_CONNECTION_STRING}"
          container: "datastream-logs"
      "@|Out-File -FilePath config\config.yaml -Encoding utf8

    • Upload to blob storage

      az storage blob upload-batch `
        --account-name datastreamconfig `
        --destination datastream-config `
        --source config

4. Deploy DataStream​

Code Deployment​

  1. Download the DataStream Web deployment package -

    Invoke-WebRequest -Uri "https://download.datastream.example.com/latest/datastream-webapp.zip" -OutFile "datastream-webapp.zip"

  2. Deploy using the Azure CLI -

    az webapp deployment source config-zip `
      --resource-group your-resource-group `
      --name your-app-name `
      --src datastream-webapp.zip

5. Configure App Service for DataStream​

  1. Update WebJobs configuration (if needed) -

    • Create a settings.job file with appropriate cron expressions
    • Upload to the wwwroot directory

  2. Configure health check endpoint -

    • Go to Health check in the App Service menu
    • Set path to /api/health
    • Enable the health check feature

6. Verify Deployment​

  1. Check deployment status in the Azure Portal:

    • Navigate to your App Service > Deployment Center
    • Verify deployment status is successful

  2. View application logs -

    • Go to App Service > Monitoring > Log stream
    • Check for successful startup messages

  3. Test the HTTP endpoint -

    $headers = @{
      "Content-Type" = "application/json"
    }
    $body = @{
      message = "test log message"
    }|ConvertTo-Json

    Invoke-RestMethod -Uri "https://your-app-name.azurewebsites.net/api/logs" -Method Post -Headers $headers -Body $body

Scaling and Performance​

Optimize your App Service deployment with:

  1. Auto-scaling rules:

    • Navigate to Scale out (App Service plan)
    • Configure rules based on CPU percentage, memory usage, or schedule
    • Example: Add an instance when CPU > 70% for 10 minutes

  2. Performance monitoring:

    • Enable Application Insights for detailed performance metrics
    • Configure alerts for abnormal conditions

  3. Instance count settings:

    • Minimum instances: 1 or more (for production workloads)
    • Maximum instances: Based on expected peak load

Networking Considerations​

For App Service deployments:

  1. VNet Integration: Enable for secure connections to:

    • Azure SQL, Redis, Service Bus, or other PaaS services
    • On-premises resources via VPN Gateway
    • Restrict outbound traffic using NSGs

  2. Private Endpoints: Configure for:

    • Secure connections from other Azure services
    • Protection from public internet exposure

  3. IP Restrictions: Limit access to specific IP ranges

Continuous Deployment​

Set up CI/CD for your DataStream App Service:

  1. GitHub Actions:

    • Generate deployment profiles in the App Service portal
    • Add secrets to your GitHub repository
    • Create workflows for automated builds and deployments

  2. Azure DevOps Pipelines:

    • Connect your repository to Azure DevOps
    • Create build and release pipelines
    • Configure automated testing and staging environments

This deployment approach provides a balance of convenience and control, with minimal infrastructure management overhead.