http://www.microsoft.com/en-us/download/details.aspx?id=44618
This week Microsoft released a number of security updates this week to patch an issue with schannel as described in this article: When the update is installed to a server running Microsoft SQL Server (So far, confirmed as issue with SQL Server 2008 R2, SQL Server 2012, SQL Server 2014) client applications that access the database via ODBC such as Microsoft Access clients pointing to SQL Tables encounter a major performance hit.
https://technet.microsoft.com/library/security/MS14-066
Our customers are reporting that this security update causes MAJOR performance problems in any Microsoft Access application with a SQL Server backend (any version). For example, a simple operation such as clicking from one line of an order to another (without performing ANY data updates) can take from 5 to 15 seconds! For users having to update hundreds of lines of orders, the application becomes nearly unusable – an activity that used to take 5 minutes could take hours.to complete.
Please, if you have not installed this update yet – DO NOT INSTALL IT to the SQL Server machine (it can be installed to clients or other servers).
If you have installed the update and are experiencing this issue, please remove the following specific update from the computer running Microsoft SQL Server to get your system back to performing normally:
Click Start, then type: “Update”, click “View installed updates”
In the list of updates scroll down until you find the list of updates installed recently (your exact date may differ). Select “Security Update for Microsoft Windows (KB2992611)” and click Uninstall. After a reboot application performance restores to normal.
As of November 13th at 11:49 AM we know that if you select each one of these under this group, and uninstall, then reboot that performance returns to normal.
KB3010788, KB3008627, KB3006226, KB3005607, KB3003743, KB3003057, KB3002885, KB2993958, KB2992611, KB2991963, KB2978120
Update: as of 2014-11-13 3:20PM our most recent test looks like it may only be KB2992611 as the root cause of this performance problem. The client machine does not appear to have to have the update removed.
Update: as of 2014-11-13 5:00PM we have a case open with Microsoft to figure out the underlying performance problem. Obviously we want our customers to be able to install security updates, but it can’t be at the expense of being able to use the software that runs your company. I’ll post additional updates when we hear back from the Windows Team.
Update: as of 2014-11-14 1:30PM: Update from Microsoft on our open case: The SQL Team is working directly with the Windows team and have been able to reproduce performance issues. They’ve created a specific tool to gather performance stats related to the issue and are working with one of my techs to gather the stats in our lab environment with both the patch installed and with it removed.
Update as of 2014-11-17 2:00 PM: Microsoft has completed data capture of several traces of client to server communication with the patch applied and removed. Status: Ball in Microsoft’s court. Waiting on analysis.
Update: Per the user comments below, this performance issue may affect any client (not just MSAccess clients) that happen to use the built-into-Windows SQL Server Driver:
Put differently, you may be affected if your connection string looks like this:
Driver={SQL Server};Server=ComputerNameRunningSQL;Database=SQLDBName;Uid=SQLAuthUser;
Pwd=SQLAuthPass;
Or like this:
Driver={SQL Server};Server=ComputerNameRunningSQL;Database=SQLDBName;
Trusted_Connection=Yes;
Update: 2014-11-28 1PM – Just received this notification e-mail from Microsoft:
The following bulletins have undergone a major revision increment.
* MS14-066 – Critical
Bulletin Information:
=====================
MS14-066 – Critical
– https://technet.microsoft.com/library/security/ms14-066
– Reason for Revision: V2.0 (November 18, 2014): Bulletin revised to announce the reoffering of the 2992611 update to systems running Windows Server 2008 R2 and Windows Server 2012. The reoffering addresses known issues that a small number of customers experienced with the new TLS cipher suites that were included in the original release. Customers running Windows Server 2008 R2 or Windows Server 2012 who installed the 2992611 update prior to the November 18 reoffering should reapply the update. See Microsoft Knowledge Base Article 2992611 for more information.
– Originally posted: November 11, 2014
– Updated: November 18, 2014
– Bulletin Severity Rating: Critical
– Version: 2.0
We have not tested this yet, but will shortly.
UPDATE: 2014-11-18 1:50PM: Part 2 of the Rev 2 patch worked!
In my example, I went back to the URL: https://technet.microsoft.com/library/security/ms14-068, selected the Windows Server 2008 R2 patch (for the test machine I was using this time I needed that one), then when I clicked download I now see a second option:
Windows6.1-KB3018238-x64.msu
On installing that 2nd file listed the problem was resolved. Performance is back to normal for this particular Windows Server 2008 R2 server with SQL installed!
UPDATE: 2014-12-01: Windows 7 SP1 No patch available.
If you are running SQL Server on your laptop for example to demo your application or as a developer we’ve discovered that there is no KB3018238 available. Attempting to install that KB on Windows 7 simply results in an error: The update is not applicable to your computer:
I’ve contacted Microsoft under my original case on this issue. Their reply so far: “KB3018238 is not available in Windows 7.” – Yes, that would be the point of my call…
So for those of you who do development on Windows 7, or those of you who are salespeople with copies of your company’s ERP database installed locally are in the same situation – either do not install KB2992611 and have a working application, or do not install it, have a working application but be at risk.
Also kb3006226 changes the behavior of certain commands dealing with redimensioning of arrays, something every programmer does.
you’re wrong. kb2992611 COMPLETELY BREAKS ODBC for remote access. What is most misleading about it is the fact that you can install it then restart MSSQLSERVER service and all seems to be ok. reboot the server and now your remote clients cant find the server and you sit back thinking “i haven’t changed anything”
What a mission it was to locate then resolve this problem. Hopefully your post and my reply will save some people the headaches we’ve been through.
Anybody else experiencing this? I’m seeing plenty of “patch now” but this is the only place I’ve seen any “hold off”….
Exactly the same problem here with KB2992611. Thank you for your post ! Please post your future MS case feedbacks here again.
So here’s the problem. This is a vulnerability that the clock is ticking on. The exploits I’ve seen nail RDP without anyone authenticating to RDP. So if you have anything that listens over a SSL protected connection, it will be possible for an attacker to take remote control of that server/workstation. Thus it’s a big fat hairy deal that you patch it as soon as you can.
Prioritize this patch. Any Exchange server – patch. Any Remote Desktop Services server – patch. If this access is not exposed over the web you have more time, but bottom line this could be nasty – therefore the urge to patch as soon as possible.
Hopefully Microsoft is also working over the weekend to get to the bottom of this side effect and can get it fixed asap.
This is good advice. Patch whatever you can. Microsoft’s Windows and SQL teams are working on the case I created. Hope they have a fix soon.
Has anybody checked whether the affected servers ( and it does all appear to be servers so far) have unwittingly set 2977292 to only allow TLS 1.2 and are using EAP authentication
I am wondering if the problems we are seeing this month with MS14-066 are a direct result of last month’s http://support.microsoft.com/kb/2977292 where an incorrect value has been applied to the registry or TLS1.2 only has been specified as default and clients are only using TLS 1.0 or 1.1
Thank you Darren!
After having our IT spend the entire Friday investigating our application, we found your article. It was a weekend saver!
Darren
Any update from Microsoft regarding your SQL performance issues
As of 9:45AM Monday, November 17th we’re still waiting on another callback. We’ve been playing phone tag with them over the weekend. Our techs are following up again this morning.
Hi Darren, hope you’re well 😉
I ran into this issue in my own environment and received the hint (thanks, Sylvain Lafontaine) to play around with other drivers and indeed, my tests were successful:
Instead of SQL Server (SQLOLEDB) use SQL Server Native Client 10.0 (SQLNCLI10) to connect to SQL 2008 R2 and performance with KB2992611 installed will be back to normal. I haven’t tested the combinations SQL 2012 / SQLNCLI11 and above.
I had the same though this weekend. Since the Access apps are communicating with the server using the Built-into-Windows version of the ODBC Driver for SQL Server, what would happen if you switched to using the “Ships with SQL Server” Native drivers? Presumably they wouldn’t be affected by the operating system patches since they don’t ship with the operating system. Unfortunately that approach won’t work in the short term for my apps because switching to the native SQL drivers would break my app in different ways that will take too much time to fix. But it may be a solution for some of the readers of this blog – so thanks for the suggestion Peter!
http://www.zdnet.com/microsoft-warns-of-problems-with-schannel-security-update-7000035835/#ftag=RSSc6e0c33
This does not appear to have affected our Access clients working against a server running 2005SP4 with the above patch. Luckily.
I saw a similar issue with a set of w2k8 servers that I manage. Can you try restarting or the Base Filtering Engine service and see if there is any effect?
thanks for this post it explains a hell of a lot of weird stuff that was happening this morning after installing that patch last night….!
Thanks Darren for posting this. Glad I found your posting before we started to roll-out this patch.
Hi, is it possible to post the MSFT SR#. And just for clarification. Is this issue gone if you disable the 4 new cipher suites (like the article posted on ZDNet) or is this totally unrelated and an additional issue?
Severity Rating: Critical
Revision Note: V2.0 (November 18, 2014): Bulletin revised to announce the reoffering of the 2992611 update to systems running Windows Server 2008 R2 and Windows Server 2012. The reoffering addresses known issues that a small number of customers experienced with the new TLS cipher suites that were included in the original release. Customers running Windows Server 2008 R2 or Windows Server 2012 who installed the 2992611 update prior to the November 18 reoffering should reapply the update.
Summary: This security update resolves a privately reported vulnerability in the Microsoft Secure Channel (Schannel) security package in Windows. The vulnerability could allow remote code execution if an attacker sends specially crafted packets to a Windows server.
=========
Can folks test and see if v2 of the update helps? If it does not holler back please.
Tests on the first of our Lab machines (a Windows Server 2008 R2 server) worked!
This is not just on SQL server. It the LSASS.EXE process across the board. We are experiencing very high LSASS issues on servers used strictly for IIS. The SQL team should not be involved here.
We suffer from this too just iis and CPU strain has doubled by LSAS.EXE.
Not funny.
Has everyone impacted installed the v2 version of this update and rebooted?
So, the SQL performance problems with this patch were caused by the addition of the cipher suites? Not actually anything to do with the security issue fixed in the patch?! It seems that way given that the “new” patch apparently just removes the cipher suites. If so, that should teach MS a lesson about bundling in non-essentials.
Also, since we’re still on 2003R2–which didn’t receive the new cipher suites–we never saw the problem, so that also fits with the above.
I reapplied the 2nd version of 2992611 and 3018238 and my SqL server and ODBC still failed every other day. I think I’m going to uninstalled 2992611 for now. When they finally find a fix please let me know. thanks
tlee – I need you to work with me to open a support case. Microsoft thinks they have the issues fixed. email me at susan-at-msmvps.com (change the -at- to @) and I can set up a support case.
I use Windows 2008R2 SP1 with DC Role) Exchange 2010 and SQL2005 on one machine. After installed KB2992611 no SQL connection is allowed. i have tried many, but now solution.
it is frustrating!
The name of the native client on SQL 2005.was “SQL Native Client” or SQLNCLI. Try one of these.
I’m trying to work out a continuous Schannel error spamming the event logs of a system, and that seems to be traced back to this same update. The curious thing is that we’re on Server 2012 R2, and the updates I’ve downloaded keep telling me they’re not applicable when I try to install them. Windows8.1-KB2992611-x64.msu is the name of the file, which by the way doesn’t show up from Windows Update when I have the system reach out to the web. Anyone else encounter this issue? Here’s the source of the file too:
https://www.microsoft.com/en-us/download/details.aspx?id=44643
Sure looks like it’s intended for Server 2012 R2 to me! Hmm.