Azure Blob Storage in ASP.NET Core

On By Anup HosurIn asp.net core

Table of Contents

  1. Overview of Azure Blob Storage
    • Types of Blobs
    • Use Cases for Blob Storage
  2. Hot Storage vs Cold Storage
    • Hot Tier
    • Cool Tier
    • Archive Tier
    • When to Use Hot, Cool, and Archive Storage
  3. CLI Commands for Creating Storage Accounts and Blob Containers
    • Creating a Resource Group
    • Creating a Standard Storage Account
    • Creating a Blob Container
    • Retrieving the Storage Account Connection String
  4. Uploading and Downloading Files in ASP.NET Core
    • Setting Up the Connection String
    • ASP.NET Core Setup for Azure Blob Storage
    • Uploading a File to Azure Blob Storage
    • Downloading a File from Azure Blob Storage
    • Best Practices for File Management
  5. Best Practices for Working with Azure Blob Storage
    • Securing Access (Key Vault, Environment Variables)
    • Retry Policies for Transient Failures
    • Multipart Upload for Large Files
    • Using SAS Tokens for Secure Access
    • Cost Optimization: Hot, Cool, Archive Tiers
  6. Shared Access Signatures (SAS) for Secure Access
    • Generating SAS Tokens via Azure CLI
    • Using SAS Tokens Programmatically in ASP.NET Core
  7. Implementing Multipart Upload for Large Files
    • Breaking Down Files into Chunks
    • Uploading Chunks to Blob Storage
    • Best Practices for Handling Large Files
  8. Versioning and Soft Delete for Data Protection
    • Enabling Versioning and Soft Delete via Azure CLI
    • Managing Blob Versions and Soft-Deleted Blobs
  9. Azure Storage Explorer
    • Using Azure Storage Explorer for Managing Blobs
    • Uploading and Downloading Files Using the Tool
  10. Lifecycle Management Policies
    • Overview of Lifecycle Management
    • Common Scenarios: Automatic Tiering, Data Expiration
    • Defining a Lifecycle Policy (with JSON Example)
    • Applying Lifecycle Policies via Azure CLI
    • Best Practices for Lifecycle Management
  11. Real-World Use Cases
    • Media Hosting (Images, Videos)
    • Big Data Analytics
    • Backup and Disaster Recovery
  12. Cost Optimization Strategies
    • Using Appropriate Tiers
    • Automating Data Tiering with Lifecycle Management
    • Monitoring Storage Costs and Usage Patterns
  13. Monitoring and Logging
    • Using Azure Monitor for Blob Storage
    • Tracking Metrics and Setting Alerts
  14. Conclusion
    • Key Takeaways for Using Azure Blob Storage Effectively
    • Summary of Best Practices and Tools

1. Overview of Azure Blob Storage

Azure Blob Storage is an object storage solution designed for storing large amounts of unstructured data such as text, binary data, media files, logs, and backups. It provides high availability, redundancy, and access control. Azure Blob Storage is essential for applications that need scalable, reliable storage for large volumes of data.

Types of Blobs:

  • Block Blobs: Best for storing text and binary data. Ideal for documents and media files.
  • Append Blobs: Optimized for scenarios where data is appended frequently, such as logging.
  • Page Blobs: Designed for frequent read/write operations, typically used for Virtual Hard Disk (VHD) files.

Use Cases for Blob Storage:

  • Hosting images and videos for web and mobile applications.
  • Storing data for big data analytics and machine learning.
  • Archiving backup and disaster recovery data.

2. Hot Storage vs Cold Storage

Hot Storage:

  • Purpose: For data that is accessed frequently.
  • Latency: Low latency and high throughput.
  • Cost: Higher storage cost but lower access costs.
  • Use Cases: Real-time applications, active data that requires frequent updates.

Cool Storage:

  • Purpose: For data that is infrequently accessed.
  • Latency: Slightly higher latency than Hot.
  • Cost: Lower storage cost compared to Hot, higher access cost.
  • Use Cases: Backups, archival, and infrequently accessed data.

Archive Storage:

  • Purpose: For data that is rarely accessed and long-term storage.
  • Latency: Retrieval can take several hours.
  • Cost: Very low storage cost but high access cost.
  • Use Cases: Regulatory data retention, long-term archival of old data.

When to Use Hot, Cool, and Archive Storage:

  • Use Hot for active data that requires quick access.
  • Use Cool for infrequent access data to save on storage costs.
  • Use Archive for data that is rarely accessed and can tolerate retrieval delays.

3. CLI Commands for Creating Storage Accounts and Blob Containers

Creating a Resource Group:

az group create --name myResourceGroup --location eastus

Creating a Standard Storage Account:

az storage account create \
  --name mystorageacct \
  --resource-group storagecontainer \
  --location eastus \
  --sku Standard_LRS \
  --kind StorageV2

Creating a Blob Container:

az storage container create \
  --name mycontainer \
  --account-name mystorageacct \
  --public-access off

Retrieving the Storage Account Connection String:

az storage account show-connection-string \
  --name mystorageacct \
  --resource-group myResourceGroup

4. Uploading and Downloading Files in ASP.NET Core

Uploading a File to Azure Blob Storage:

 public async Task<string> UploadAsync(Stream stream,string fileName)
 {
     var stopWatch = new Stopwatch();
     stopWatch.Start();
     var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
     await containerClient.CreateIfNotExistsAsync();
     var blobClient = containerClient.GetBlobClient(fileName);
     await blobClient.UploadAsync(stream, true);
     stopWatch.Stop();
     logger.LogInformation($"Uploaded {fileName} in {stopWatch.ElapsedMilliseconds} ms");
     return blobClient.Uri.ToString();
 }

Downloading a File from Azure Blob Storage:

 public async Task<Stream> DownloadAsync(string fileName)
 {
     var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
     var blobClient = containerClient.GetBlobClient(fileName);
     var response = await blobClient.DownloadAsync();
     return response.Value.Content;
 }

Best Practices for File Management:

  • Secure Access: Use environment variables or Azure Key Vault to manage connection strings.
  • Optimize Performance: Implement retry policies and use chunked uploads for large files.
  • Cost Efficiency: Move blobs to appropriate storage tiers based on their access patterns.

5. Best Practices for Working with Azure Blob Storage

Securing Access:

  • Use Azure Key Vault or environment variables to securely manage connection strings.
  • Implement Identity-based Access Control using Azure Managed Identity.

Retry Policies for Transient Failures:

  • Use Polly library to implement retry policies for transient errors.

Multipart Upload for Large Files:

  • Split large files into chunks and upload them in parallel to improve performance and reliability.

Using SAS Tokens for Secure Access:

  • Generate Shared Access Signatures (SAS) to provide limited, time-bound access to specific blobs.

Cost Optimization:

  • Choose the right storage tier based on access frequency.
  • Regularly review and move blobs to appropriate tiers using lifecycle management.

6. Shared Access Signatures (SAS) for Secure Access

Generating SAS Tokens via Azure CLI:

az storage blob generate-sas \
  --account-name mystorageacct \
  --container-name mycontainer \
  --name <blob-name> \
  --permissions r \
  --expiry <expiry-date-time> \
  --https-only

Using SAS Tokens Programmatically in ASP.NET Core:

public string GenerateSasToken(string blobName)
{
    var blobServiceClient = new BlobServiceClient("<connection-string>");
    var blobContainerClient = blobServiceClient.GetBlobContainerClient("<container-name>");
    var blobClient = blobContainerClient.GetBlobClient(blobName);

    var sasBuilder = new BlobSasBuilder
    {
        BlobContainerName = blobContainerClient.Name,
        BlobName = blobClient.Name,
        Resource = "b",
        ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
    };

    sasBuilder.SetPermissions(BlobSasPermissions.FromString("r")); // Read permissions

    return blobClient.GenerateSasUri(sasBuilder).ToString();
}

7. Implementing Multipart Upload for Large Files

Breaking Down Files into Chunks:

  • Split large files into smaller chunks to handle uploads more efficiently.

Uploading Chunks to Blob Storage:

public async Task UploadLargeFileAsync(IFormFile file)
{
    var blobContainerClient = _blobServiceClient.GetBlobContainerClient(_containerName);
    var blobClient = blobContainerClient.GetBlobClient(file.FileName);

    var blockBlobClient = blobClient.GetBlockBlobClient();
    var blockIds = new List<string>();
    var blockSize = 4 * 1024 * 1024; // 4 MB

    using (var stream = file.OpenReadStream())
    {
        var buffer = new byte[blockSize];
        int bytesRead;
        int blockNumber = 0;

        while ((bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
        {
            var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockNumber.ToString("d6")));
            blockIds.Add(blockId);

            using (var blockStream = new MemoryStream(buffer, 0, bytesRead))
            {
                await blockBlobClient.StageBlockAsync(blockId, blockStream);
            }

            blockNumber++;
        }

        await blockBlobClient.CommitBlockListAsync(blockIds);
    }
}

Best Practices for Handling Large Files:

  • Parallel Uploads: Upload multiple chunks in parallel to improve performance.
  • Error Handling: Implement retry logic to handle any network or service interruptions.

8. Versioning and Soft Delete for Data Protection

Enabling Versioning and Soft Delete via Azure CLI:

az storage account blob-service-properties update \
  --account-name mystorageacct \
  --resource-group myResourceGroup \
  --enable-versioning true

az storage account blob-service-properties update \
  --account-name mystorageacct \
  --resource-group myResourceGroup \
  --delete-retention <days>

Managing Blob Versions and Soft-Deleted Blobs:

  • Blob Versioning: Maintain multiple versions of a blob to recover from accidental overwrites.
  • Soft Delete: Recover blobs that were accidentally deleted within the retention period.

9. Azure Storage Explorer

Using Azure Storage Explorer for Managing Blobs:

  • Azure Storage Explorer is a GUI tool that allows you to easily manage and interact with Azure Blob Storage.

Uploading and Downloading Files Using the Tool:

  • Uploading: Drag and drop files into the blob container to upload.
  • Downloading: Right-click on a blob and select the download option to save it locally.

10. Lifecycle Management Policies

Overview of Lifecycle Management: Lifecycle Management Policies automate the movement of blobs between storage tiers and data deletion based on predefined rules.

Common Scenarios:

  • Automatic Tiering: Move blobs to cooler tiers as they age.
  • Data Expiration: Automatically delete blobs that are no longer needed.

Defining a Lifecycle Policy (with JSON Example):

{
  "rules": [
    {
      "name": "MoveToCool",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 365
            }
          }
        },
        "filters": {
          "blobTypes": ["blockBlob"]
        }
      }
    }
  ]
}

Applying Lifecycle Policies via Azure CLI:

az storage account management-policy create \
  --account-name mystorageacct \
  --policy @policy.json

Best Practices for Lifecycle Management:

  • Test Policies: Validate policies with a sample dataset before applying them broadly.
  • Monitor Usage: Use Azure Monitor to track and optimize your lifecycle policies based on actual data usage.

11. Real-World Use Cases

Media Hosting:

  • Use Blob Storage to host and deliver images and videos for web and mobile applications.

Big Data Analytics:

  • Store large datasets for analysis and processing in big data environments.

Backup and Disaster Recovery:

  • Archive backup data and ensure disaster recovery with cost-effective storage options.

12. Cost Optimization Strategies

Using Appropriate Tiers:

  • Match storage tiers with data access patterns to balance performance and cost.

Automating Data Tiering with Lifecycle Management:

  • Implement policies to automatically move data to the appropriate storage tier based on its usage.

Monitoring Storage Costs and Usage Patterns:

  • Use Azure Monitor to track usage and adjust storage strategies based on actual needs.

13. Monitoring and Logging

Using Azure Monitor for Blob Storage:

  • Track important metrics such as storage capacity, request success rates, and data egress using Azure Monitor.

Tracking Metrics and Setting Alerts:

  • Set up alerts for anomalies such as increased error rates or unexpected cost spikes.

14. Conclusion

Azure Blob Storage offers a versatile and scalable solution for managing unstructured data. By understanding and leveraging the different storage tiers, lifecycle management policies, and best practices, you can optimize your data storage for performance and cost. Integrating Blob Storage with ASP.NET Core Web APIs enables robust file management capabilities, while tools like Azure CLI and Azure Storage Explorer enhance your ability to manage and monitor storage efficiently.

Leave a comment