SQL Server upgrade from 2005 to 2008 R2 – more notes

I have already put down some notes about upgrading SQL Server from 2005 to 2008 R2, but since it’s been a few weeks I gained some useful experience. I’ll divide it into two parts – what works and what doesn’t, so it’s pretty much like a standard lessons learnt document. It’s all pretty fresh – I had two upgrades in May, both successful, but not without hassle (self-inflicted, of course).

What works:

  1. Copying installation source locally instead of installing from a network share – network disruptions have no impact on upgrade. I had my upgrade at the same time as LAN maintenance and it had a minimal impact on SQL Server upgrade.
  2. Having a local account with administrative rights – in case you can’t use built-in Administrator account you might consider creating a local account and adding it to local Administrators group as it will free you from network/login issues. Just remember to clean up afterwards.
  3. Upgrading from SQL Server 2005 databases in a compatibility level 90 – straightforward, never met a single one that had problems after setting compatibility level to 100.
  4. Migrating of Reporting Services between SQL Servers (both 2005) by means of backup\restore – yes, but in 2005 I had to install SSRS, initialize its database and overwrite it on restore. Never tried it in 2008+, but I presume it works exactly the same. When I tried to restore to a new database, and then reconfigure SSRS, it wasn’t able to initialize. However, there’s a trick with deleting encryption keys which might also work in this case – worth checking in free time.

What doesn’t work

  1. Upgrading a read-only database – renders it unusable. I did it (nothing to be proud of) and to fix it I had to put it in emergency mode, detach, reattach and set it back to read-only. Please note it’s what I did and worked for me  – checked with DBCC CHECKDB afterwards and the database was fine. Why emergency mode, you might ask – the database startup fails during recovery phase, because it requires to be upgraded as server version has changed while it’s read-only. So it leaves you with a 2008 R2 server using databases with internal version 661, but a read-only database remains with internal version 611 (because it’s read-only), so the server issues an error that data files are inaccessible and you can’t even detach it. The emergency mode is used here to suppress recovery and allow detaching – I wasn’t brave enough to try setting the database to read-write after emergency mode on – also worth checking in free time if it works and what happens to the database. Alternative – restore latest backup (since the database ia read-only, it should not change) to another SQL Server 2005 box, set it to read-write and copy to 2008 R2 instance. Lucky me as the whole database was read-only – if it were only a filegroup inside, that would be a bigger issue (see Paul Randal’s post at SQLMag.com about it).
  2. Upgrading SQL Server 2000 databases to 2008 R2 – some of them contain some deprecated features like old-style joins and it turns out that you can upgrade the server, but have to leave the database on compatibility level 80. I’ve seen this happen a few times, and the only solution is to review database structure and remove all the deprecated features. The thing is – you might not always have time or resources to do it, but it’s where SQL Server Upgrade Advisor comes in, the reports can help you track where the problems can be.
  3. This is me grumbling about a thing that is by design – a number of restarts required to upgrade your SQL Server, especially on Windows Server 2003. If you didn’t reboot since last patching session, restart. If you don’t have Windows Installer 4.5 (and I don’t since I’m starting from 2005), next restart. Upgrade SQL Server, next restart. After Service Pack, another restart. Now multiply it by a number of instances on the server that are upgraded, add some time for some preparations before, clean up after, plan for some contingency, and you can barely squeeze two instances in 6 hours. Luckily this isn’t one of the new HP ProLiant DL380 G7 servers that take 10 minutes to reboot, spending 9 minutes on power and thermal calibration. This is also an important point – when planning your maintenance window, consider all the activities. If the installation takes 30 minutes and updates another 15, both require a restart which takes 15 minutes, plus one upfront, then your downtime is not 45 minutes, but 90! It is crucial because if you miscalculate, you might not fit in the maintenance window and then you have a service down, not to mention that if something goes wrong, you need some time for rollback plan (which cannot fail – but imagine that tape drives are busy with your 70%-completed Exchange backup with still 2 TB to go and you have to wait till it’s done to restore 10 GB database).

Now, I might have another set comments pretty soon – a round of upgrades is waiting for me in June plus a retirement. If you have any comments or questions, feel free to share.

SQL Server 2005 SP3 to 2008 R2 upgrade – planning and issues

As part of my DBA activities I was tasked with performing feasibility study of upgrade of SQL Server 2005 to 2008 R2, which produced a nice roadmap I’m using now to perform step-by-step migrations. I thought it was going to be quite smooth, but sometimes – mostly due to inconsistencies in environment – issues happen, unfortunately.

First of all, if you happen to lead a project of such migration be sure to run SQL Server Upgrade Advisor to identify blocking issues early and solve them in advance. This may include old-style joins using *= or =* (if you have compatibility model 80 databases). Most SQL Server 2005 and later instances are compatible so it should be quite straightforward, but do yourself a favour, check it and be on a safe side.

Second, when you’re approaching a migration, clean the environment up as much as possible. Here’s my case – I had a server where databases had no owner, so I fixed it. But then it turned out that DEP is prevented installation of SQL Server from remote source. If you’re on Windows Server 2003, chances are that you need .NET 3.5 and Windows Installer 4.5 and it requires additional restart (that’s what happened to me). When I managed to run SQL Server Setup, final upgrade checks returned “SQL Server health check” failed. This was a bit tricky, because it required my account to get “Debug programs” privilege on local machine (careful! it needs logout), which took another 30 minutes.

Consider the restarts also take time. You might need 2 or 3 restarts, if going from 2005 SP3 to 2008 R2 SP1 and if you’re running servers like HP ProLiant DL380G7 which are quick, but take awful lot of time to boot, you may count 30 minutes for those restarts. 30 minutes in 2 hours maintenance window is 25% and it’s really a lot when a quarter of downtime you’re off and waiting for server to come up.

If you’re responsible for upgrade, have some time on side for such unexpected situations – if you think a maintenance window of 2 hours is sufficient, consider to have 4 hours booked. This will give you time to solve things if they appear without great time pressure and even if you happen to have a total failure that you need to do restore, you’re still on time.

Third, do all the backups needed. Take a backup of user databases and master prior to backup, take one after you complete upgrade and if you apply further SPs and CUs, backup after them as well. If you use a tape drive, do a full backup of your whole server after you complete migration. And don’t forget to verify the backup when you take it.

All those tips could be briefly wrapped up in a PM proverb “if you fail to plan, you plan to fail”. You can’t argue with that kind of wisdom.

Database mirroring pseudorandom ramblings

I liked database mirroring since it appeared in SQL Server 2005. In fact, it’s the only HA technique I feel comfortable with (apart from it’s obvious limitations). It’s neither as blunt as log shipping, as random as replication nor as complex as clustering. It either works or it doesn’t – and you know it straight away after you set it up – you don’t have to wait for snapshot to be generated, sent over the network and applied on a subscriber or for a backup to be copied and restored on your secondary.

The incentive for this post was one by Glenn Berry, which held one answer to my mirroring-related questions. The database mirroring is considered deprecated in SQL Server 2012; it’s going to be available in 2012 and the next version, but then it will be removed from SQL Server and replaced by AlwaysOn Availability Groups. Given the releases of new major versions of SQL Server my totally wild guess it would be like… 2018? Later? All in all, it’s not the major point of concern right now, but something worth remembering and planning when time comes.

What gives me second thoughts is the second part of deprecation note saying “If your edition of SQL Server does not support AlwaysOn Availability Groups, use log shipping”. What I understand is – combining it with new SQL Server licensing model – you can get replication and log shipping in Standard Edition, clustering and AlwaysOn in Enterprise Edition, but what about mirroring? I would like it to remain in Standard Edition, but to be 100% sure I asked on MSDN forums and I’m waiting for an answer. On the other hand, if it were to go to Enterprise Edition, I would expect dramatic costs increase due to per-physical-core licensing model, which would not be very welcome by most organizations.

When it comes to facts about database mirroring apart from standard source of knowledge (Books On-Line) I also recommend the following:

  1. The forementioned post and it’s sequel by Glenn Berry – a detailed introduction and complete checklist of points to be considered while implementing database mirroring.
  2. Pro SQL Server 2008 Mirroring – a book by Robert Davis and Ken Simmons, which apparently is not only about mirroring, but also about other high availability techniques as well.
  3. Blog posts by Robert Davis tagged with database mirroring – lots of additional information.
  4. An MCM video followed by a demo presented by Paul Randal – to give an insight how the mirroring works internally.
  5. Microsoft KB article 2001270 on things to consider while implementing mirroring, mentioning exact Performance Objects to watch out for in case of performance issues.

Now few words why I like mirroring in SQL Server:

  1. It works.
  2. It allows you to do automatic client redirection in case of failure (provided you have a high safety scenario with a witness) – see an example of implementation, works with SQL Native Client, ADO.NET and SQL Server JDBC driver.
  3. You can run it on Standard Edition and you don’t need extra hardware for that (unlike clustering).
  4. In high safety mode, you can have zero data loss (unless your mirroring is broken).
  5. You can have it working even if you are not in a domain, or you don’t have trust relationship between domains in which mirroring partners are – I have recently presented an example during local SQL Server user group meeting based on Books On-Line example available here. It’s completely opposite to simple scenario in one domain when it comes to implementation, but I recommend trying that one out.

For the other side of the coin – things to remember when implementing database mirroring:

  1. There was only one Redeemer! 🙂 you can have only one mirror server – this limitation is overcome in 2012’s AlwaysOn, where you can have 4 replicas, but it has it’s costs.
  2. Mirror is not easily accessible but you can use snapshots to query it.
  3. Server-level objects are not transferred with mirroring – you have to take care of logins, Database Mail profiles, SQL Server Agent jobs or Maintenance plans yourself. See Books On-Line article on jobs and logins.
  4. Mirroring requires full recovery model so you have to be absolutely sure about the transaction log of mirrored databases. Index maintenance, for example, can kill mirroring performance if you have 100GB of log records related to index reorganize sent to the mirror (and it will kill you twice – first the network, then the mirror server).
  5. In SQL Server 2008 and 2008 R2 FILESTREAM and mirroring don’t work together. You can’t have both in one database. To be checked – SQL Server 2012’s FileTables and mirroring.
  6. The failover is not instant – see Paul Randal’s post and Books On-Line. In short – it depends on the length of your REDO queue, so if you’re not monitoring it, then please do, just for the sake of not being surprised in the event of failover.
  7. The failures not detected automatically – there are four cases here (again, I’m quoting Paul Randal):
    • SQL Server unavailability – detected in a second
    • Machine unavailability – default is 10 seconds, you may increase it using ALTER DATABASE SET PARTNER TIMEOUT
    • Unavailable log drive – it takes 20 seconds for a warning, and next 20 seconds (total 40 seconds) for database to be taken offline
    • Page corruption – might not even be detected 😦
  8. It’s not the witness who performs the failover, but the mirror! The mirror and the witness agree that the principal cannot be contacted and the mirror switches to a principal. If you don’t believe, set three Profiler sessions on all three mirroring participants and watch out for ALTER DATABASE statements when failover occurs.

Wrapping it all up, database mirroring is a really useful high availability technique (especially when combined with other methods – be it clustering or replication). It is easy to implement, does not have extra hardware requirements. Though it’s marked as deprecated in SQL Server 2012, it is going to be available and useful for at least couple of years, but it has it’s downsides which should be carefully considered when planning.