Skip to main content
Version: 1.2.0

Deployment: On Azure Functions

This guide explains how to deploy DataStream as a serverless application using Azure Functions. This event-driven deployment model offers automatic scaling to zero when idle, consumption-based pricing, and minimal infrastructure management.

Benefits​

Deploying DataStream on Azure Functions provides distinct advantages:

  • Serverless architecture: No server management or capacity planning required
  • Pay-per-execution: Only pay for the compute resources you use
  • Automatic scaling: Scales based on workload, including scaling to zero
  • Event-driven processing: Natively respond to events from various sources
  • Simplified deployment: Focus on code rather than infrastructure

When to Use Azure Functions​

Azure Functions deployment is ideal for:

  • Variable workloads with periods of inactivity
  • Event-driven processing scenarios
  • Microservice architectures where components need independent scaling
  • Cost-sensitive deployments where consumption-based pricing is preferred
  • Lightweight processing of streaming data

Azure Functions Hosting Plans​

Select the appropriate hosting plan based on your needs:

PlanDescriptionBest For
ConsumptionPay-per-execution with automatic scalingVariable workloads, cost optimization
PremiumEnhanced performance, pre-warmed instancesConsistent workloads, no cold starts
App ServiceRun on dedicated VMsPredictable workloads, existing App Service plans

For most DataStream deployments, the Premium plan offers the best balance of performance and flexibility.

Deployment Steps​

1. Prepare Azure Resources​

  1. Create a Function App in the Azure Portal:

    • Select .NET 6 as the runtime
    • Choose your preferred hosting plan (Premium recommended)
    • Enable Application Insights for monitoring
    • Configure storage account for function state

  2. Create supporting resources:

    • Storage Account: For configuration and state management
    • Event Hub or Service Bus (optional): For message buffering
    • Key Vault: For secure storage of credentials

2. Configure Function Settings​

  1. Add application settings in the Function App Configuration:

    ParameterSetting
    DATASTREAM_CONFIG_MODEazure_blob
    DATASTREAM_STORAGE_CONNECTIONYour storage connection string
    DATASTREAM_CONFIG_CONTAINERdatastream-config
    DATASTREAM_CONFIG_PATHDirector/config/
    DATASTREAM_PROCESSOR_MODEfunctions

  2. Configure Function-specific settings:

    • Set Function runtime version to ~4
    • Configure host keys and CORS settings as needed
    • For Premium plan: Set minimum instance count (1+ for production)

3. Prepare Function Code​

Deploy from Visual Studio​

  1. Create a new Azure Functions project - Use Visual Studio or VS Code with Azure Functions extension. Select .NET 6 as the runtime.

  2. Add the DataStream NuGet package:

    <PackageReference Include="DataStream.Functions" Version="1.0.0" />

  3. Create HTTP Trigger function:

    using System.Threading.Tasks;
    using Microsoft.AspNetCore.Mvc;
    using Microsoft.Azure.WebJobs;
    using Microsoft.Azure.WebJobs.Extensions.Http;
    using Microsoft.AspNetCore.Http;
    using Microsoft.Extensions.Logging;

    public static class DataStreamProcessor
    {
      [FunctionName("ProcessLogs")]
      public static async Task<IActionResult> Run(
          [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
          ILogger log)
      {
        // Initialize DataStream processor
        var processor = new DataStreamFunctionProcessor();
        
        // Process incoming request
        var result = await processor.ProcessHttpRequestAsync(req);
        
        // Return response
        return new OkObjectResult(result);
      }
    }

  4. Publish to Azure using Visual Studio's publish functionality

4. Create DataStream Configuration​

  1. Prepare configuration files -

    • Create config directory

      New-Item -ItemType Directory -Force -Path config

    • Create main config file

      @"
      triggers:
        - name: http_trigger
          type: http
          route: logs
          methods: ["POST"]
          auth_level: function

      processors:
        - grok:
            field: message
            patterns:
              - "%{COMMONAPACHELOG}"
        - set:
            field: parsed_timestamp
            value: "{{{timestamp}}}"

      targets:
        - name: azure_storage
          type: azure_blob
          connection_string: "${AZURE_STORAGE_CONNECTION_STRING}"
          container: datastream-logs
          path_format: "{date}/{hour}"
      "@|Out-File -FilePath config\functions.yaml -Encoding utf8

  2. Upload to blob storage -

    az storage blob upload-batch `
      --account-name youraccount `
      --destination datastream-config `
      --source config

5. Set Up Function Bindings​

For optimal performance, configure appropriate bindings:

  1. HTTP Trigger for receiving log data

    • Configure authentication level (function, admin, or anonymous)
    • Set appropriate route templates

  2. Queue output binding for buffering messages:

    [FunctionName("BufferLogs")]
    public static async Task Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = "logs")] HttpRequest req,
        [Queue("log-buffer")] IAsyncCollector<string> outputQueue,
        ILogger log)
    {
      // Process and buffer logs
    }

  3. Event Hub trigger for processing batched events:

    [FunctionName("ProcessEvents")]
    public static async Task Run(
        [EventHubTrigger("logs", Connection = "EventHubConnection")] EventData[] events,
        ILogger log)
    {
      // Process events in batches
    }

6. Verify Deployment​

  1. Test the function endpoint:

    $headers = @{
      "Content-Type" = "application/json"
      "x-functions-key" = "YOUR_FUNCTION_KEY"
    }
    $body = @{
      message = "192.168.1.1 - user1 [10/Oct/2023:13:55:36 -0700] ""GET /index.html HTTP/1.0"" 200 2326"
    }|ConvertTo-Json

    Invoke-RestMethod -Uri "https://your-function-app.azurewebsites.net/api/logs" -Method Post -Headers $headers -Body $body

  2. Check function logs:

    • Navigate to your Function App > Functions > ProcessLogs > Monitor
    • View invocation logs and traces

  3. Monitor output storage - Verify that processed logs are being written to the configured storage location.

Scaling and Performance​

Optimize Azure Functions deployment with:

  1. Premium plan settings:

    • Configure minimum instance count (1+ for production)
    • Set maximum burst limit based on expected peak load
    • Enable pre-warmed instances to eliminate cold starts

  2. Consumption plan optimization:

    • Implement retry logic for cold start handling
    • Keep functions warm with scheduled pings if needed
    • Set appropriate timeout values

  3. Performance tuning:

    • Configure batch size for optimal throughput
    • Implement connection pooling for database connections
    • Use async patterns throughout your code

Cost Optimization​

For Azure Functions deployments, control costs with:

  1. Right-sizing: Choose the appropriate plan based on your workload pattern
  2. Execution optimization: Minimize function execution time and memory usage
  3. Batching: Process multiple events in a single function execution
  4. Scheduled scaling: For predictable workloads, schedule scaling rules

Limitations and Considerations​

Be aware of these Azure Functions constraints:

  1. Execution time limits:

    • Consumption plan: 10-minute maximum execution
    • Premium plan: 30-minute maximum execution

  2. Network constraints:

    • Limited support for custom TCP/UDP listeners
    • HTTP/HTTPS primary ingress method

  3. State management:

    • Functions are stateless by design
    • Use external storage for state persistence

  4. Cold start impact:

    • First requests after idle periods may experience latency
    • Premium plan reduces cold start issues