Showing posts with label sql. Show all posts
Showing posts with label sql. Show all posts

Wednesday, December 14, 2011

Optimize your database cursors (considering SQL Azure)

Yeah, I know most of the DBAs (if not all) say to avoid using cursors in your SQL Server code, but there are still some things, which you can only achieve via cursors. You can read a lot discussions on whether to use cursors or not, is it good, is it bad.

My post is not about arguing what is good and what is bad. My post is about a tiny little option, which, if your logic allows you can use to optimize how your cursor works.

So we are using cursors, for good or bad. Everything might work just fine if we are using on-premise SQL Server, and if the server is not under heavy load. Our stored procedures, which are using cursors are executing in a matter of seconds. There is nothing unusual. We deploy our application to The Cloud. And of course we utilize SQL Azure as our DB backend. Now strange things begin happening. Our stored procedures crash with timeout exceptions. If we login to the server and use the good “sp_who3” (yes, this works in SQL Azure!) to see the processes running, we notice that some procedures do report a  SOS_SCHEDULER_YIELD. You can read a lot of information on what does that mean. As by definition:

Occurs when a task voluntarily yields the scheduler for other tasks to execute. During this wait the task is waiting for its quantum to be renewed.

Most of the resources you will find explaining what does lot of SOS_SCHEDULER_YIELD mead, will suggest high CPU load, non-optimized queries, etc. But we look at our code and there is nothing unusual. Also, as this is SQL Azure, we can’t see the actual CPU load of the OS. We can’t add more CPU or more RAM. What do we do now ?

Well, review once again our cursor logic! If it is the case that we only read from the cursor’s data. We only read forward, never backward. We never change cursor’s data (update/delete). Then there is a pretty good chance that we can use the FAST_FORWARD keyword when declaring our cursors:

Specifies a FORWARD_ONLY, READ_ONLY cursor with performance optimizations enabled. FAST_FORWARD cannot be specified if SCROLL or FOR_UPDATE is also specified.

It is amazing performance booster and load relief! And we, most probably, will never see again the SOS_SCHEDULER_YIELD process status for our procedures.

Most (if not all) of the cursors I’ve written are never reading backward or updating data, so I pretty amazed to see the performance differences using this keyword. I for sure will use it from now on, whenever possible!.

Wednesday, March 23, 2011

SQL Azure limitations

During my talks on SQL Azure, and the session I’ve been to (only locally in Bulgaria) we always listen limitations like “excessive resource usage”, “long running transactions”, “idle time”. But it is very hard to find out officially what are the exact numbers behind these statements. Now that I prepare for my session next week at Microsoft Developer Days 2011 I am hunting for a numbers. And here they are (keep in mind that these numbers are subject to change at any time without notification):

    • Excessive Memory Usage: When there is memory contention, sessions consuming greater than 16-megabyte (MB) for more than 20 seconds are terminated in the descending order of time the resource has been held, such as the oldest session is terminated first. Termination of sessions stops as soon as the required memory is available.When the connection is lost due to this reason, you will receive error code 40553.
    • Idle Connections: Connections to your SQL Azure database that are idle for 30 minutes or longer will be terminated. Since there is no active request, SQL Azure does not return any error.
    • Transaction Termination: SQL Azure kills all transactions after they run for 24 hours. If you lose a connection due to this reason, you will receive error code 40549.
    • Lock Consumption: Sessions consuming greater than one million locks are terminated. When this happens, you will receive error code 40550. You can query the sys.dm_tran_locks dynamic management view (DMV) to obtain information about the current state of locking in SQL Azure.
    • Log File Size: Transactions consuming excessive log resources are terminated. The maximum permitted log size for a single transaction is 1-gigabyte (GB). When the connection is lost due to this reason, you will receive error code 40552.

Full list of limitations and a very good reading: http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-connection-management-in-sql-azure.aspx

Friday, June 6, 2008

Data Mining by Rafal Lukawiecki

Microsoft has recently published a series of video sessions on Data Mining by Rafal Lukawiecki on their TechNet Spotlight.

This is, in my opinion, a vast and interesting subject which was partially covered by Rafal when he came to Sofia on 25th of January this year. As he promised a series of video sessions are already on the TechNet Spotlight, so anyone who hadn't chance to be here and listen to Rafal live, can go and watch the videos online. Once more - really, really amazing subject and incredible speaker!

Sunday, April 6, 2008

Microsoft SQL Server or Oracle Database Server ?

It's the question that drives discussions for decades. Recently I ended up with very interesting article. It's called "The real difference between SQL Server and Oracle".

The Intro:

For years now there's been a constant war between Microsoft supporters and Oracle supporters. Oracle has these features, SQL Server has these features, etc. But that's not really where the real importance lies. Sure, functionality is a part of it because your database should be able to do what you need it to do. However, do you want to know what the real difference between the two companies is and why Microsoft has made such a strong impact in the industry?

You can read the full story here. I hope you also find it interesting :)

Saturday, January 5, 2008

MySQL - my (Un)Favourite Database

In my last project I decided to go out with MySQL. Why - do not ask why ;)

Anyway.

In the process of building, debugging, developing I noticed that some of the searches does not return a correct result. A search in a VARCHAR field (for Cyrillic characters). I looked for the table encoding - it was UTF8; I looked for the column encoding - it was UTF8 (both collations utf8_general_ci); I looked for the server default encoding - UTF8; I looked for the encoding passed by the client application - again it was UTF8.

Suddenly, by incident I noticed - the encoding of the database was cp1251 (the Cyrillic one). It killed me.

A simple command solved the problem:

ALTER DATABASE {db_name} DEFATULT ENCODING utf8, DEFAULT COLLATION utf8_general_ci;

But I was inserting correct values, when I was browsing the table I could see the right values, but when I search for specific values in the STORED PROCEDURE with AD-HOC SQL built dynamically in the procedure, I could not get the results ...

No Idea ... but I begin to look more and more to the MS SQL server ... and its EXPRESS edition...

Thursday, December 20, 2007

SQL, Joins, Unions and more .. continued II

Last night I decided to let MySQL think about that query ... It seemed to have completed calculations in the morning. Not surprisingly it has returned the very same data like did the MS SQL - The multiplication of the results. But it has taken 6604156 milliseconds (~6604 seconds) to obtain the result. The same result that MS SQL EXPRESS returned in 274709 milliseconds (~275 seconds).

No further comments.
Case closed.

Wednesday, December 19, 2007

SQL, Joins, Unions and more .. continued

Hi again.

Today I made a test - transferred the data into MS SQL Server 2005 Express SP 2.0.

Guess What ? :)

The situation is almost the same ! Wondering why the result was messed, the execution time was arround '274709' - I guess this is the "ms" - "duration" column in the MS SQL Server Profiler. Which is arround 5 minutes.
The mess I am talking about is the following:
The situation is that in of the tables I have about 120k records, which should be displayed, on the other one I have about 5k records. Insted of showing "120000" for "di_count", and "5000" for "ei_count", in the result I got '594163480' (which is the number of rows in one of the left joined tables multiplied by the number of rows in the other one) for each of them - the same figure for the two columns. What is that ?

I just want to make a select from one table, than join second table, then join third table to the FIRST one, then join the last table AGAIN to THE FIRST one. Why is it joining the result alltogether?

Anyway .. using UNION, VIEW and work with the VIEW solves the problem, but I was just wondering why this happens ...

I j ust can't help showing you these two screens:

For the MSSQL Profiler:

mssql

And for the MySQL (note: the execution is still not finished!):

mysql

I have no comment!

Tuesday, December 18, 2007

SQL, Joins, Unions and more

Ok, this is will be about MySQL once again.

I had that problem. Let's assume we have 3 tables:
1. smetki
2. di_invoices
3. dk_invoices
4. ei_invoices

Each of the XX_invoices has a "foreign_key" to PK of "smetki". I am using MyISAM.
If I perform the following 3 queries separately:
---------------------------------------------------------
select
s.smetka_id, s.created_on,
count(di.invoice_id) as di_count
from
smetki as s
left outer join di_invoices as di on (s.smetka_id = di.smetka_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id

-----------------------------------------------------------
select
s.smetka_id, s.created_on,
count(dk.invoice_id) as dk_count
from
smetki as s
left outer join dk_invoices as dk on (s.smetka_id = dk.smetka_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id

------------------------------------------------------------
select
s.smetka_id, s.created_on,
count(ei.invoice_id) as ei_count
from
smetki as s
left outer join ei_invoices as ei on (s.smetka_id = ei.smetki_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id
------------------------------------------------------------
I have no problem, I have proper indexes and I got the results for less then 200 ms from each query.
But I need the 3 results into 1 single resultset so I decided to make a join like that:
select
s.smetka_id, s.created_on,
count(di.invoice_id) as di_count,
count(dk.invoice_id) as dk_count,
count(ei.invoice_id) as ei_count
from
smetki as s
left outer join di_invoices as di on (s.smetka_id = di.smetka_id)
left outer join dk_cor_invoices as dk on (s.smetka_id = dk.smetka_id)
left outer join ei_invoices as ei on (s.smetka_id = ei.smetki_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id

Now I have query which executes more than 15 minutes !! I couldn't wait to the end and killed it. There must be something wrong somewhere?
I sould notice that for the given period there are only 6 rows in the "smetki" table, and different number of rows in each xx_invoices table that correspond to the whole resultset from "smetki". I have to mention that there is one intersection between two of the XX_invoices tables where there are about 120k rows for one "smetka_id", and for the same "smetka_id" there are about 5k rows in the other.
Also - there are no other queries running by the time of measuring the performance.

What could be the solution. A friend of mines helped me. Make a UNION he said. So I did it like that:

(select
s.smetka_id, s.created_on,
count(di.invoice_id) as di_count, 0 as dk_count, 0 as ei_count
from
smetki as s
left outer join di_invoices as di on (s.smetka_id = di.smetka_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id)
UNION
(select
s.smetka_id, s.created_on,
0 as di_count, count(dk.invoice_id) as dk_count, 0 as ei_count
from
smetki as s
left outer join dk_invoices as dk on (s.smetka_id = dk.smetka_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id)
UNION
(select
s.smetka_id, s.created_on, 0 as di_count, 0 as ei_count,
count(ei.invoice_id) as ei_count
from
smetki as s
left outer join ei_invoices as ei on (s.smetka_id = ei.smetki_id)
where
s.created_on between '2007-12-10' and '2007-12-15'
group by
s.smetka_id )

OK, but now I have 3 rows for each single "smetka_id" - one row for the results from each table. And then we decided to make a VIEW over that UNION.

Gues what. Now I have the simple query:

select smetka_id, sum(di_count), sum(dk_count), sum(ei_count) from v_test group by smetka_id

Which returns me the desired result for less than a second !

I hope this would help your daily work with MySQL and SQL in general.

Wednesday, October 31, 2007

MySQL Tips & Tricks /MAX_ROWS & AVG_ROW_LENGTH/

Hello again ;)
I ran into a tricky situation last couple of days. I have some big tables with varchar fields where I have to JOIN ON varchar fields. The problem does not concern only the JOIN - it is a general problem using varchar / text fields used to be searched one way or another.

So - the situation: We have one table with about 1M rows (relatively small amount data for each row ~90 bytes) on one hand. And another table with records to be matched - about 100k rows (relatively medium amount of data for each row ~ 500 bytes). We do a SELECT which JOINs the two tables based on VARCHAR columns. Do not ask "why?" - This is the business logic; it cannot be otherwise (at least in the beginning). If we index the columns we will got the results fast! What is fast? Less than 50 ms to get around 200 joined rows. Well, OK, you got me - it depends strongly on the hardware configuration and MySQL settings. Let's assume this is our limit. I was very happy to discover that it works that fast. The speed is critical, because we will need to join around 100k rows.

Where is the problem then? Well, it happened that the first table is populated with another several hundred thousand records and it became with about 1.9M rows. And? And, it appeared that the MySQL should copy the temp table on the hard drive. As you can guess - the result is: 204 joined rows (not 204k, but only two hundred and four) for ..... 417 seconds. What about 100k rows? Killing me.

What can we do about that? Well, I know (the business logic predicts) that the first table - that with about 2M rows would never have more than 4 million rows. I also noticed that partitioning is only partially supported in the beta (RC yet) of MySQL 5.1.x, but the recommended for production environment is 5.0.x which does not support partitioning. So, I discovered the great options MAX_ROWS and AVG_ROW_LENGTH. Which does what - it says to the MySQL engine, that this table would never have more than MAX_ROWS tows, and the average row length would be AVG_ROW_LENGTH. You can find the average row length by executing the command SHOW TABLE STATUS FROM [DB_NAME] LIKE 'TBL_NAME'. So, if you know, if your business logic predicts that you will have a large table, BUT this table will have relatively known content (like it will have not more than 4M rows), and you know the average row length, you can set the MAX_ROWS and AVG_ROW_LENGTH for that table. The AVG_ROW_LENGTH is required if you have blob/text/varchar fields.

That will cause the MySQL engine to optimize the space required for that table, also optimizing the myisam_data_pointer_size, which sets the number of bytes used for internal row pointers. Yes - it works only with MyISAM tables. But you will be surprised. After I set the MAX_ROWS and AVG_ROW_LENGTH on that table with the 2M records - I got a surprising results - 90k joined rows for ..... 47 seconds !!

I think I will use these options often in the future, what about you?