SessionAffinity plugin for Windows Azure

In previous post I've reviewed what Session Affinity is and why it is so important for your Windows Azure (Cloud Service) deployments. I also introduced the SessionAffinity and SessionAffinity4 plugins, part of the Azure Plugin Library project. Here I will describe what this plugin is and how it works.

The SessionAffinity plugin is based around Microsoft's Application Request Routing module, which can be installed as an add-on in Microsoft's web server – IIS (Internet Information Services). This module has dependency of the following other (useful) modules:

• URL Rewrite – similar to the Apache's mod_rewrite. You can even translate most of the Apache's mod_rewrite rule to IIS URL Rewrite Rules;
• Web Farm Framework - simplifies the provisioning, scaling, and management of multiple servers;
• ARR - enables Web server administrators, hosting providers, and Content Delivery Networks (CDNs) to increase Web application scalability and reliability through rule-based routing, client and host name affinity, load balancing of HTTP server requests, and distributed disk caching;
• External Cache

The two most important features of ARR that will help us achieve Session Affinity are the URL Rewrite and load balancing. Of course they only make sense when there is a Web Farm of Servers to manage.

Here is a basic diagram which illustrates what happens to your [non-.NET-deployment] when you use SessionAffinity Plugin:

First and most important of all – SessionAffinity plugin only works with Worker Roles! This type of role you will use when you want to deploy a non .NET web server (Apache, Apache Tomcat, NGIX, etc.) . This is very important. Web Role is special kind of Role, and the internal Azure infrastructure does additional things on IIS configuration which literally mess with the Session Affinity plugin and ARR configurations. So, use only Worker Roles when you want to use Session Affinity.

The plugin itself consists of two main modules:

Installer bootstrapper – takes care of installing the ARR module and all its dependencies

SessionAffinityAgent.exe – a .NET based console application which is both configuration utility and watchdog service. It completes the initial configuration of the ARR – sets the load balance algorithm to Weighted Round Robin and configures the affinity based on cookie! The second important job of this application is to monitor the Azure Environment for changes via the RoleEnvironment.Changed event. This event occurs when any change to the role environment happens – instances are added or removed, configuration settings is changed and so on. You can read more about handling Role Environment changes on this excellent blog post. When you happen to add more role instances (or remove any) all the ARR modules on all the instances must be re-configured to include all the VMs in the Web Farm. This is what Session Affinity agent is doing by constantly monitoring the environment.

With this setup now there is ARR module installed on each of the instances. Each ARR module knows about how many total server are there. There is also a software load balancer (part of the Web Farm framework), which also knows which are all the servers (role instances).

These are the components in a single instance:

Web requests going to port 80 are accepted by the local IIS site. It has a configured URL Rewrite Rule which transfers the request to the local Web Farm Framework. Web farm framework is aware of all the servers configured in the setup (all role instances). It checks where an affinity cookie exists in the requests. If such cookie does not exists, random server is chosen from the pool and new cookie is created to keep track of which user was assigned to the user. The request is finally redirected internally to the Apache listening on port 8080. This information is synchronized across all servers that are part of the Web Farm. Next request will have the cookie and the user will be sent to the same server.

Here is simple flow diagram for a web request that goes on public port 80 to the cloud service deployed with Session Affinity plugin:

SessionAffinity4 plugin (the one that works with Windows Server 2012 / OS Family 3) has one configurable option:

Two10.WindowsAzure.Plugins.SessionAffinity4.ArrTimeOutSeconds

As its name suggest this is Timeout in seconds. Timeout for what? If we look at the flow chart we will see that there are a lot of things to happen. The last one is waiting for response from the Apache Web server (on port 8080). This timeout indicates how long will the ARR module wait for response from Apache before sending Timeout Error (HTTP 50x). If you don't set value for this setting, 180 seconds (3 minutes) is considered a reasonable time to wait. If you want, you can change this value in the Service Configuration file. For example if you have some heavy pages or long running operations you may want to increase the timeout. Be careful, the error page returned to the end user is a standard HTTP 500 error page! So it is better that your server never times-out, or at least you have to configure the value of ArrTimeOutSeconds to a value which is greater then the expected longest processing page.

Session Affinity and Windows Azure

Everybody speaks about recently announced partnership between Microsoft and Oracle on the Enterprise Cloud. Java has been a first-class citizen for Windows Azure for a while and was available via tool like AzureRunMe even before that. Most of the customers I've worked with are using Apache Tomcat as a container for Java Web Applications. The biggest problem they face is that Apache Tomcat relies on Session Affinity.

What is Session Affinity and why it is so important in Windows Azure? Let's rewind a little back to this post I've written. Take a look at the abstracted network diagram:

So we have 2 (or more) servers that are responsible for handling Web Requests (Web Roles) and a Load Balancer (LB) in front of them. Developers has no control over the LB. And it uses one and only one load balancing algorithm – Round Robin. This means that requests are evenly distributed across all the servers behind the LB. Let's go through the following scenario:

• I am web user X who opens the web application deployed in Azure.
• The Load Balancer (LB) redirects my web request to Web Role Instance 0.
• I submit a login form with user name and password. This is second request. It goes to Web Role Instance 1. This server now creates a session for me and knows who I am.
• Next I click "my profile" link. The requests goes back to Web Role Instance 0. This server knows nothing about me and redirects me to the login page again! Or even worse – shows some error page.

This is what will happen if there is no Session Affinity. Session Affinity means that if I hit Web Role Instance 0 first time, I will hit it every time after that. There is no Session Affinity provided by Azure! And in my personal opinion, Session Affinity does not fit well (does not fit at all) in the Cloud World. But sometimes we need it. And most of the time (if not all cases), it is when we run a non-.NET-code on Azure. For .NET there are things like Session State Providers, which make developer's life easier! So the issue remains mainly for non .net (Apache, Apache Tomcat, etc).

So what to do when we want Session Affinity with .NET web servers? Use the SessionAffinity or SessionAffinity4 plugin. This basically is the same "product", but the first one is for use with Windows Server 2008 R2 (OS Family = 2) while the second one is for Windows Server 2012 (OS Family = 3).

I will explain in a next post what is the architecture of these plugins and how exactly they work.

Active Directory in Azure – Step by Step

Ever since Windows Azure Infrastructure Services were announced in preview I keep hearing questions "How to run Active Directory in Azure VM? And then join other computers to it". This article assumes that you already know how install and configure Active Directory Directory Services Role, Promote to Domain Controller, join computers to a Domain, Create and manage Azure Virtual Networks, Create and manage Azure Virtual Machines and add them to Virtual Network.

Disclaimer: Use this solution at your own risk. What I describe here is purely my practical observation and is based on repeatable reproduction. Things might change in the future.

The foundation pillar for my setup is the following (totally mine!) statement: The first Virtual Machine you create into an empty Virtual Network in Windows Azure will get the 4th IP Address in the sub-net range. That means, that if your sub-net address space is 192.168.0.0/28, the very first VM to boot into that network will get IP Address 192.168.0.4. The given VM will always get this IP Address across intentional reboots, accidental restarts, system healing (hardware failure and VM re-instantiating) etc., as long as there is no other VM booting while that first one is down.

First, lets create the virtual network. Given the knowledge from my foundation pillar, I will create a virtual network with two separate addressing spaces! One addressing space would be 192.168.0.0/28. This will be the addressing space for my Active Directory and Domain Controller. Second one will be 172.16.0.0/22. Here I will add my client machines.

Next is one of the the most important parts – assign DNS server for my Virtual Network. I will set the IP Address of my DNS server to 192.168.0.4! This is because I know (assume) the following:

• The very first machine in a sub-network will always get the 4th IP address from the allocated pool;
• I will place only my AD/DC/DNS server in my AD Designated network;

Now divide the network into address spaces as described and define the subnets. I use the following network configuration which you can import directly (however please note that you must have already created the AffinityGroup referred in the network configuration! Otherwise network creation will fail):

<NetworkConfiguration 
  xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <Dns>
      <DnsServers>
        <DnsServer name="NS" IPAddress="192.168.0.4" />
      </DnsServers>
    </Dns>
    <VirtualNetworkSites>
      <VirtualNetworkSite name="My-AD-VNet" AffinityGroup="[Use Existing Affinity Group Name]">
        <AddressSpace>
          <AddressPrefix>192.168.0.0/29</AddressPrefix>
          <AddressPrefix>172.16.0.0/22</AddressPrefix>
        </AddressSpace>
        <Subnets>
          <Subnet name="ADDC">
            <AddressPrefix>192.168.0.0/29</AddressPrefix>
          </Subnet>
          <Subnet name="Clients">
            <AddressPrefix>172.16.0.0/22</AddressPrefix>
          </Subnet>
        </Subnets>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

Now create new VM from gallery – picking up your favorite OS Image. Assign it to sub-net ADDC. Wait to be provisioned. RDP to it. Add AD Directory Services server role. Configure AD. Add DNS server role (this will be required by the AD Role). Ignore the warning that DNS server requires fixed IP Address. Do not change network card settings! Configure everything, restart when asked. Promote computer to Domain Controller. Voilà! Now I have a fully operations AD DS + DC.

Let's add some clients to it. Create a new VM from gallery. When prompted, add it to the Clients sub-net. When everything is ready and provisioned, log-in to the VM (RDP). Change the system settings – Join a domain. Enter your configured domain name. Enter domain administrator account when prompted. Restart when prompted. Voilà! Now my new VM is joined to my domain.

Why it works? Because I have:

• Defined DNS address for my Virtual Network to have IP Address of 192.168.0.4


• Created dedicated Address Space for my AD/DC which is 192.168.0.0/29


• Placed my AD/DC designated VM in its dedicated address space


• Created dedicated Address Space for client VMs, which does not overlap with AD/DC designated Address Space


• I put client VMs only in designated Address Space (sub-net) and never put them in the sub-net of AD/DC

Of course you will get same result if with a single Address Space and two sub-nets. Being careful how you configure the DNS for the Virtual Network and which sub-net you put your AD and your Client VMs in.

This scenario is validated, replayed, reproduced tens of times, and is being used in production environments in Windows Azure. However – use it at your own risk.

Windows Azure Basics–Compute Emulator

Following the first two posts of the series “Windows Azure Basics” (general terms, networking) here comes another one. Interestingly enough, I find that a lot of people are confused what exactly is the compute emulator and what are these strange IP Addresses and port numbers that we see in the browser when launching a local deployment.

If you haven’t read the Windows Azure Basics – part 2 Networking, I strongly advise you to do so, as rest of current post assumes you are well familiar with real Azure deployment networking components.

A real world Windows Azure deployment has following important components:

• Public facing IP Address (VIP)
• Load Balancer (LB) with Round Robin routing algorithm
• Number of Virtual Machines (VM) representing each instance of each role, each with its own internal IP address (DIP – Direct IP Address)
• Open ports on the VIP
• Open ports on each VM

In order to provide developers with as close to real world as possible, a compute emulator needs to simulate all of these components. So let's take a look what happens when we launch locally a Cloud Service (a.k.a. Hosted Service).

VIP Address

The VIP address for our cloud service will be 127.0.0.1. That is the public IP Address (VIP) of the service, via which all requests to the service shall be routed.

Load Balancer

Next thing to simulate is the Azure Load Balancer. There is a small software emulated Load Balancer, part of the Compute Emulator. You will not see it, you are not able to configure it, but you must be aware of its presence.  It binds to the VIP (127.0.0.1). Now the trickiest thing is to find the appropriate ports to bind. You can configure different Endpoint for each of your roles. Only the Input Endpoints are exposed to the world, so only these will be bound to the local VIP (127.0.0.1). If you have a web role, the default web port is 80. However, very often this socket (127.0.0.1:80) is already occupied on a typical web development machine. So, the compute emulator tries to bind to the next available port, which is 81. In most of the cases port 81 will be free, so the "public" address for viewing/debugging will be http://127.0.0.1:81/. If port 81 is also occupied, compute emulator will try the next one – 82, and so on, until it successfully binds to the socket (127.0.0.1:XX). So when we launch a cloud service project with a web role we will very often see browser opening this wired address (http://127.0.0.1:81). The process is same for all Input Endpoints of the cloud service. Remember, the Input endpoints are unique per service, so an Input Endpoint cannot be shared by more than one Role within the same cloud service.

Now that we have the load balancer launched and bound to the correct sockets, let's see how the Compute Emulator emulated multiple instances of a Role.

Web Role

Web Roles are web applications that run within IIS. For the web roles, compute emulator uses IIS Express (and can be configured to use full IIS if it is installed on the developer machine).  Compute Emulator will create a dedicated virtual IP Address on the local machine for each instance of a role. These are the DIPs of the web role. A local DIP looks something like 127.255.0.0. Each local "instance" then gets the next IP address (i.e. 127.255.0.0, 127.255.0.1, 127.255.0.2 and so on). It is interesting that the IP Addresses begin at 0 (127.255.0.0). Then it will create a separate web site in IIS Express (local IIS) binding it to the created Virtual IP Address and port 82. The emulated load balancer will then use round robin to route all requests coming to 127.0.0.1:81 to these virtual IP Addresses.

Note: You will not see the DIP virtual address when you run ipconfig command.

Here is how does my IIS Express look like when I have my cloud service launched locally:

Worker role

This one is easier. The DIP Addressing is the same, however the compute emulator does not need IIS (neither IIS Express). It just launches the worker role code in separate processes, one for each instance of the worker role.

The emulator UI

When you launch a local deployment, Compute Emulator and Storage Emulator are launched. You can bring the Compute Emulator UI by right clicking on the small azure colored windows icon in the tray area:

For purpose of this post I've created a sample Cloud Service with a Web Role (2 instances) and a Worker Role (3 instances). Here is the Compute Emulator UI for my service. And if I click on "Service Details" I will see the "public" addresses for my service:

Known issues

One very common issue is the so-called port walking. As I already described, the compute emulator tries to bind to the requested port. If that port isn't available, it tries next one and so on. This behavior is known as "port walking". Under certain conditions we may see port walking even between consequent runs of same service – i.e. the first run compute emulator binds to 127.0.0.1:81, the next run it binds to 127.0.0.1:82. The reasons vary, but the obvious one is "port is busy by another process". Sometimes the Windows OS does not free up the port fast enough, so port 81 seems busy to the compute emulator. It then goes for the next port. So, don't be surprised, if you see different ports when debugging your cloud service. It is normal.

Another issue is that sometimes browser launches the DIP Address (http://127.255.0.X:82/) instead the VIP one (http://127.0.0.1:81/). I haven't been able to find a pattern for that behavior, but if you see a DIP when you debug your web roles, switch manually to the VIP. It is important to always use our service via the VIP address, because this way we also test out application cloud readiness (distributing calls amongst all instances, instead of just one). If the problem persists, try restarting Visual Studio, Compute Emulator or the computer itself. If issue still persists, open a question at StackOverflow or the MSDN Forum describing the exact configuration you have, ideally providing a Visual Studio solution that constantly reproduces the problem. I will also be interested to see the constant repeatable issue.

Tip for the post: If you want to change the development VIP address ranges (so that it does not use 127.0.0.1) you can check out the following file:

%ProgramFiles%\Microsoft SDKs\Windows Azure\Emulator\devfabric\DevFC.exe.config

DevFC stands for "Development Fabric Controller". But, please be careful with what you do with this file. Always make a backup of the original configuration before you change any setting!

Happy Azure coding!

Bending the Windows Azure Media Services–H.264 Baseline profile

Disclaimer: What I will describe here is not officially supported by Microsoft and by Windows Azure Media Services. This means that if task fails you cannot open support ticket, nor you can complain. I discovered these hidden feature by digging deeply into the platform. Use the code and task preset at your own risk and responsibility. And note that what works now, may not work tomorrow.

Exploring the boundaries of Windows Azure Media Services (WAMS), and following questions on StackOverflow and respective MSDN Forums, it appears that WAMS has previously supported H.264 Baseline Profile and have had a task preset for Baseline Profile. But now it only has Main Profile and High Profile task presets. And because the official documentation says that Baseline Profile is supported output format, I don’t see anything wrong in exploring how to achieve that.

So what can we do, to encode a video into H.264 baseline profile if we really want? Well, use the following Task Preset at your own will (and risk ):

<?xml version="1.0" encoding="utf-16"?>
<!--Created with Expression Encoder version 4.0.4276.0-->
<Preset
  Version="4.0">
  <Job />
  <MediaFile
    WindowsMediaProfileLanguage="en-US"
    VideoResizeMode="Letterbox">
    <OutputFormat>
      <MP4OutputFormat
        StreamCompatibility="Standard">
        <VideoProfile>
          <BaselineH264VideoProfile
            RDOptimizationMode="Speed"
            HadamardTransform="False"
            SubBlockMotionSearchMode="Speed"
            MultiReferenceMotionSearchMode="Speed"
            ReferenceBFrames="True"
            AdaptiveBFrames="True"
            SceneChangeDetector="True"
            FastIntraDecisions="False"
            FastInterDecisions="False"
            SubPixelMode="Quarter"
            SliceCount="0"
            KeyFrameDistance="00:00:05"
            InLoopFilter="True"
            MEPartitionLevel="EightByEight"
            ReferenceFrames="4"
            SearchRange="32"
            AutoFit="True"
            Force16Pixels="False"
            FrameRate="0"
            SeparateFilesPerStream="True"
            SmoothStreaming="False"
            NumberOfEncoderThreads="0">
            <Streams
              AutoSize="False"
              FreezeSort="False">
              <StreamInfo>
                <Bitrate>
                  <ConstantBitrate
                    Bitrate="4000"
                    IsTwoPass="False"
                    BufferWindow="00:00:04" />
                </Bitrate>
              </StreamInfo>
            </Streams>
          </BaselineH264VideoProfile>
        </VideoProfile>
        <AudioProfile>
          <AacAudioProfile
            Level="AacLC"
            Codec="AAC"
            Channels="2"
            BitsPerSample="16"
            SamplesPerSecond="44100">
            <Bitrate>
              <ConstantBitrate
                Bitrate="160"
                IsTwoPass="False"
                BufferWindow="00:00:00" />
            </Bitrate>
          </AacAudioProfile>
        </AudioProfile>
      </MP4OutputFormat>
    </OutputFormat>
  </MediaFile>
</Preset>

You can quickly check whether it works for you by using the RunTask command line, part of the MediaServicesCommandLineTools project. The H264_BaselineProfile.xml is provided for reference in the etc folder of the project. You can tweak and Audio and Video bitrates at your will by editing the XML.