Another common reason for Windows Azure role to stuck in Initializing/Busy/Stopping

There is another common reason that will cause your role to stuck into that loop when you deploy it on Windows Azure. And it is common for Web and Worker Roles.

You will most probably encounter this error if you are using latest Windows Azure Tools (as you should to) and you are creating a new project (rather than upgrading your existing one).

The new project template comes with enabled and started DiagnosticsMonitor. In order for this Monitor to work properly and collect and store data it needs a Windows Azure Storage account. The account configuration is saved as Role Configuration Entry in the .CSCFG file. The default “Connection string” is this one:

    1 <Setting name="DiagnosticsConnectionString" value="UseDevelopmentStorage=true" />

Note the “UseDevelopmentStorage=true” in the value for our DiagnosticsConnectionString. If you just make a “Hello World” Web Role and you upload it to Azure environment, that role will never start. It will go a Initializing/Busy/Stopping/Stopped loop. You need to either stop the DiagnosticsMonitor or change the connection string.

Lest first see how the connection string should look like:

    7       <Setting name="StorageClientAccount"

    8                value="DefaultEndpointsProtocol=[https|http];AccountName=[your_account_name];AccountKey=[your_account_key]" />

In this “Connection string” setting for Diagnostics monitor you need to provide:

• DefaultEndpointsProtocol – http or https
• AccountName – your storage account name (endpoints are automatically built up)
• AccountKey – the authentication key that is generated at Storage management part of the portal.

Without specifying these settings your role will not run on Azure environment.

How about disabling DiagnosticsMonitor? Sure you can do so. If you have a WebRole there will be a single file defining a Class (usually called WebRole.cs) that derives from RoleEntryPoint. This class looks something like this:

   24 public class WebRole : RoleEntryPoint

   25 {

   26     public override bool OnStart()

   27     {

   28         DiagnosticMonitor.Start("DiagnosticsConnectionString");

   29 

   30         // For information on handling configuration changes

   31         // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.

   32         RoleEnvironment.Changing += RoleEnvironmentChanging;

   33         ...

It is important to note line 28, on which there is a call: DiagnosticMonitor.Start("DiagnosticsConnectionString");

If want to remove the DiagnosticsMonitor just strip the entire OnStart() method. However I suggest not to do so, because you will loose all logging functionalities of Windows Azure.

I hope that this eliminates most common reasons for headaches of Windows Azure Role stuck in Initializing/Busy/Stopping/Stopped loops.

Windows Azure role stuck in Initializing/Busy/Stopping

Recently I see a lot of questions with that description around Windows Azure Forums. There is one particular reason that appears to be causing this behaviour in most of the cases. I want to describe it with respect to Web Role and Worker Role

Web Role

If you have a web role prepared for Windows Azure deployment be sure that you double check that you have set “Copy Local” attribute to TRUE for any third party assemblies that you reference in your project. These are all assemblies that are not part of Microsoft .NET Framework 3.5 SP1. Such as Microsoft ReportViewer, Microsoft MVC, Microsoft WCF RIA Services, Unity, nHibernate, SandCastle. Any.

Worker Role

While requirements for referenced assemblies in Worker roles are same as requirements for these in Web Roles, there is another important point to observe. And this is that your Run() method should never end. If it ends, than Windows Azure Fabric thinks there is something wrong are recycles your role. That’s why you initial Run method will look like this:

    1         public override void Run()

    2         {

    3             // This is a sample worker implementation. Replace with your logic.

    4             Trace.WriteLine("WorkerRole2 entry point called", "Information");

    5 

    6             while (true)

    7             {

    8                 Thread.Sleep(10000);

    9                 Trace.WriteLine("Working", "Information");

   10             }

   11         }

It is important to not remove the while (true) cycle as it it the “lifecycle” of our worker role. If we remove it,our worker role will enter an endless loop of:

[runtime] Role entrypoint . COMPLETED OnStart()
[runtime] Role entrypoint . CALLING   Run()
Information: Worker Process entry point called
[runtime] Role entrypoint . COMPLETED Run() ==> ROLE RECYCLING INITIATED
[runtime] Role instance recycling is starting
Information: ...
Information: ...
[fabric] Role state Recycle
[fabric] Role state Stopping
[fabric] Role state Stopped
[fabric] Role state Stopping
[fabric] Role state Stopped
[fabric] Role state Suspended
[fabric] Role state Aborted
[fabric] Role state Teardown
[fabric] Role state Destroyed
[fabric] Role state Created
[fabric] Role state Suspended

Why is it so is well described by Steve Marx as answer to my question on MSDN Forums:

The model is that you're expected to never return from Run().  (If you do, we think something went wrong and restart you.)
If you have other threads doing work, you can just go into an infinite sleep at the bottom of Run().  (Thread.Sleep(Timeout.Infinite))

http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/2ce85358-b7e2-4b2b-bfbd-952b65e31fa2

I really hope that this post will save you lot of time and frustration using Windows Azure.

Windows Azure bookmarks

I decided to post a couple of very useful links to resources on Windows Azure, which I happen to use very often and almost always search for them first.

So here there are:

Register on Microsoft Connect to get CTP account (limited)
Note: it is one registration process for both Windows Azure and SQL Azure. However the invitation tokens does not come together and you have to be patient, you will recieve both tokens in couple of business days.
http://go.microsoft.com/fwlink/?LinkID=129453

Windows Azure Platform on MSDN

http://msdn.microsoft.com/en-us/azure/default.aspx

 

Tools & SDKs

Windows Azure Tools for Visual Studio (NOV'2009)
http://www.microsoft.com/downLoads/details.aspx?familyid=6967FF37-813E-47C7-B987-889124B43ABD&displaylang=en

Windows Azure SDK (NOV'2009)
http://www.microsoft.com/downLoads/details.aspx?familyid=772990DA-8926-4DB0-958F-95C1DA572C84&displaylang=en

Microsoft .NET Services SDK (NOV'CTP)
http://www.microsoft.com/downLoads/details.aspx?familyid=C80EBADF-7EB8-4A62-ABCD-0B57FA3855F8&displaylang=en

Additional Azure Samples, including ASP.NET Providers for Table Storage

http://code.msdn.microsoft.com/windowsazuresamples

Azure on Channel 9

http://channel9.msdn.com/learn/courses/Azure/

MSDN Documentation on "Online Services":

http://msdn.microsoft.com/en-us/library/dd163896.aspx

SQL Azure Firewall

http://msdn.microsoft.com/en-us/library/ee621782.aspx

Azure Platform Forums on MSDN:

http://social.msdn.microsoft.com/Forums/en-US/category/azure

Bug in Windows Azure Tools for VS2008

I just submitted a bug to Microsoft about the mess being created when you want to have both .NET RIA Services Tools for VS 2008 and Windows Azure Tools for VS2008.
The description:
When you have both Windows Azure tools and .NET RIA Services tools for Visual Studio 2008, the former will conflict the last.
In general, if you have installed only .NET RIA services tools for VS, a new type of project is added (affected one) - .NET RIA Services Library. It creates two projects - one Silverlight Library and one .NET Class library.
However, if you also have Windows Azure tools for VS installed, and want to create .NET RIA Serivces Library, the second project type will be Windows Azure Worker Role

Connecting to SQL Azure from ASP.NET

Let’s talk a little bit about SQL Azure.
Generally when you subscribe for SQL Azure at https://sql.azure.com/ you will receive your activation token (although is bit too late already to go for it, but you may have a chance). Once activated the service, it will automatically create a new project for you named “SDS-only CTP Project” and you will have options to manage it.
Then you go and create your first SQL Azure data base. From now on you will be fighting to use the toolset you used to, when developing against SQL Server. However there are some niceties you need to learn in order to get working with it.
First of all is how to connect to SQL Azure using either sqlcmd or SSMS (SQL Server Management Studio). I wrote detailed post about it here: http://blogs.staykov.net/2009/08/how-to-use-sql-azure.html
Next you will want to simply run a GridView bound to the SQL Azure database. Well … do not expect that either Server Explorer, or SQL DataSource creation wizard will work with it. Just forget about these tools for now. You are on your own. Or not exactly.
Here: SqlAzureSample.zip I have uploaded two sample projects and simple SQL script. So after you have read how-to-connect-to-sql-azure, and after you have set up your first database and assigned a login to it, you can start with the projects:

• create_insert.sql simply creates a table and inserts some records into it.
• WebSiteToSqlAzure.zip contains simple ASP.NET web-site which is bound to SQL Azure
• WebRoleToSqlAzure.zip contains a CloudService project with single web role, which is bound to SQL Azure

Both web sites that are attached are using one and the same ConnectionString:
"Server=tcp:[YOUR_SERVER].ctp.database.windows.net;Database=[YOUR_DB];User ID=[LOGIN_WITHOUT_@SERVER];Password=[PASSWORD];Trusted_Connection=False;"
There are some placeholders for you to replace:
[YOUR_SERVER] – the server your CTP of SQL Azure resides. Something like "ccmlt4qacx”
[YOUR_DB] – the database name you have created
[LOGIN_WITHOUT_@SERVER] – the login you have associated with database
[PASSWORD] – the password you have created for the login
Hope that this will be a good starting point for you!
And hope to continue Azure series :)

SqlAzure Migration tool

I was wandering wether we have to wait too long to have some tool to easily transfer our DB Schema to the cloud.
The anwser came just one day ago and it is here: http://sqlazuremw.codeplex.com/
LIghtweight, simple, easy to use, this tool generates (DROP) Create t-sql statements compatible with SqlAzure. The Scripts generated are bit large (Well it depends on your schema) and it takes some time to execute all, but it works.

I am still searching for a way to easily (or not, but automated) TRUNCATE entire DB and recreate tables again. Everything I found over internet is kind-of useless with Azure since there are no stored procedures like "sp_MSForEachTable" or so. The closest match I found is executing some statement like this:

DECLARE @TableName NVARCHAR(2048)
DECLARE @ConstraintName NVARCHAR(2048)
DECLARE Constraints CURSOR FOR
SELECT TABLE_NAME, CONSTRAINT_NAME FROM INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE
OPEN Constraints
FETCH NEXT FROM Constraints INTO @TableName, @ConstraintName
WHILE @@FETCH_STATUS = 0
BEGIN
EXEC('ALTER TABLE [' + @TableName + '] DROP CONSTRAINT [' + @ConstraintName + ']');
FETCH NEXT FROM Constraints INTO @TableName, @ConstraintName;
END
CLOSE Constraints
DEALLOCATE Constraints

DECLARE Tables CURSOR FOR
SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES
OPEN Tables
FETCH NEXT FROM Tables INTO @TableName
WHILE @@FETCH_STATUS = 0
BEGIN
EXEC('DROP TABLE [' + @TableName + ']');
FETCH NEXT FROM Tables INTO @TableName;
END
CLOSE Tables
DEALLOCATE Tables

But it works only partially on Azure and if you do not have lots tables and constraints. For some reason Azure can't drop some of the constraints.

Azure Logs

The convinient RoleManager.WriteToLog static method provides way to write event logs for your clouded application. In the reference:
http://msdn.microsoft.com/en-us/library/microsoft.servicehosting.serviceruntime.rolemanager.writetolog.aspx
you will find that the allowed values for "eventLogName" are:
Critical, Error, Warning, Information, Verbose

It's a good idea to have them handy :)

Here is a complete How-to guide for writing and reading logs:
http://blog.benday.com/archive/2008/11/07/23201.aspx

I will just add that there is a great tool at CodePlex: http://azurestorageexplorer.codeplex.com/
Which is visual storage explorer. Something that I missed is that account configuration requires restart of the application (at least for me). If you are going to wander how to configure your cloud storage - just enter your cloud storage account name (without http://, and without blob.core.windows.net, or queue.core.windows.net, etc.) and your primary or secondary key - it works with both keys.

Azure Worker Roles

Have you tried it yet ?
It is a background worker process, jsut like a windows service. So you can create your background working processes just like you used with Windows Service. It is very simple to implement:

public class WorkerRole : RoleEntryPoint
{
public override void Start()
{
// This is a sample worker implementation. Replace with your logic.
RoleManager.WriteToLog("Information", "Worker Process entry point called");
while (true)
{
Thread.Sleep(10000);
RoleManager.WriteToLog("Information", "Working");
}
}
public override RoleStatus GetHealthStatus()
{
// This is a sample worker implementation. Replace with your logic.
return RoleStatus.Healthy;
}
}

As simple as that. Of course you can have your System.Timers.Timer instance or more and schedule your processing logic to start up at regular periods.

Azure WebRole and Custom Error Handlers

A useful article on how to use custom handlers in the cloud:
http://www.shanmcarthur.net/cloud-services/azure-tips/custom-error-handlers

The main idea is that Azure is runnin on IIS7, so you have the power to plug-in into the integrated pipline mode.

In short:

<>
< errormode="Custom">
< statuscode="404">
< statuscode="404" responsemode="ExecuteURL" path="/page-not-found.aspx">
< /httpErrors >
< /system.webServer >

Put the configuration in system.webServer, insted of in system.web.

I would also add a handler for 400 status code. This is if you are making some custom http handlers.

How to use SQL Azure

Well,
If you want to play with SQL Azure it is little tricky. You cannot use SQL Server management Studio. However you can connect using ADO.NET. One option is to develop small app that does basic things like listing tables, enables editing tables etc, which actually is not that hard - to cover basic functionality.
Until then, the smoothest way I found is usign the SQLCmd tool.
Better described here: http://english.zachskylesowens.net/2009/08/18/connecting-to-sql-azure/
However this is the "quick-start":
sqlcmd -S MY_SERVER_NAME.ctp.database.windows.net -U MY_USERNAME@MY_SERVER_NAME -d DATABASE_NAME

The key here is that you must use MY_USERNAME@MY_SERVER_NAME !
For example, if your server name is h38ssfjeiwh201, your username is admin and connecting to sample_db your connection would be.
sqlcmd -S h38ssfjeiwh201.ctp.database.windows.net -U admin@h38ssfjeiwh201 -d sample_db

Also good point is that you must explicitly specify DataBase name to use, since AzureSQL CTP still does not support USE DataBaseName command.

Good luck and happy times with Sql Azure!

Azure Worker Role config settings app.config

Recently playing with Azure July CTP I stumbled upon an issue which appeared to be known.
The thing that might drive you crazy is that the Publishing wizard does not apply settings from App.Config file. So basically your deployed worker process will have no idea about configuration settings.
It seemed to be a known issue and there are some discussions over here:
solution: http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/cc11be6e-097a-499c-96c7-5089d01e2e08

other thread: http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/604384f0-9d20-4be6-a02c-24ba1ff1bf82

The solution is trivial:
You have to add an environment variable to your user account:
Key: AddAppConfigToBuildOutputs
Value: true
And you need to restart Visual Studio for the change to take place.

Apache mod_rewrite and per-host directives / subdomains

This feature of Apache is very powerful and useful. Well, for those who really understand it :) For the rest of us, it is hours of trials-and-fails until we reach wanted configuration.

So, let's say we have a domain (domain.com). We have setup a WildCard DNS records to point to our web server (*.domain.com) and we also have setup our VirtualHost with this wildcards:

ServerName domain.com
ServerAlias *.domain.com

Now, we want to make a translation of
http://sub.domain.com/ to http://www.domain.com/subdomains/sub/

Most of the articles and tutorials I found over the internet are just redirecting requests. So the end-users never actually stay on sub.domain.com, but are transferred to www.domain.com/subdomains/sub . This I don't want. Here is a brief set of rules how to achieve that: http://www.debian-administration.org/article/Wildcard_hosting_with_Apache_and_Bind
They are jsut redirecting.

However I want the end-users to stay on http://sub.domain.com/ however all internal requests to go to http://www.domain.com/subdomains/sub . And also I want it that way only that particular "sub".

And here is the simple solution:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www.domain.com$ [NC]
RewriteCond %{HTTP_HOST} ^(sub)\.domain\.com
RewriteRule ^(.*) http://www.domain.com/subdomains/sub/$1 [L,P]

The keypoint here is that is we skip the P flag, mod_rewrite will make automatic external redirect. It is not smart enough to guess that this is our server. That's why we have to put the P-flag, which claims that we want this redirection to take place via "internal Proxy"

Well, that's for now. Hope to post some new stuff soon