The Red Exclamation Points (A Suspense/Thriller Mini-Novella)…

This evening was going like any other busy evening for a Coretek employee with a deadline fast approaching.  I was working with Citrix Support to figure out what the problem is with a Citrix XenMobile (an Enterprise Mobile Device Manager), when the dreaded warning texts started pouring in…

I received message after message about “servers down, 8:30 PM”…  The nice thing about it is I figured out who was by their phones to respond to the alerts.  After a quick couple of text messages, I acknowledged the alerts and began troubleshooting. 

I knew the issue was with our VMWare Vcenter Server, and my first reaction was, “…well I’m busy let’s just reboot this thing.”  So I rebooted the server and waited for it to idle down before launching VSphere, which now showed a bunch of orphaned VMs (“orphaned” is a bad word in Vmware, essentially meaning that a portion of the vm be it disk or configuration files can’t be located). 

So my mind starts racing to a network or storage issue.  I go to click on an orphaned VM and vSphere Crashed.  I now *Face Palm* myself (…sigh…) and go back to the vCenter server and attempt to start the service “VMware VirtualCenter Server.”  It starts, and as I open vSphere again, and it almost immediately crashes. 

So now I’m starting to get really nervous, and I’m thinking I’m going to need to do a restore; meanwhile, the support engineer from Citrix is on the line asking me to show her my Netscaler configuration and AppController configs.  I tell her, “I’ll show you what I have in a minute, ½ of my network just alerted me it is down.”  I believe it was one of those “lost in translation” moments because she waits and a minute later says, “well can I see the configs?”.  I give in show her the configs and run for another laptop from my desk to continue troubleshooting.  And as I log in and open Event Viewer on the vCenter server to see what is going on…  All I see are red exclamation points…

 

Red Exclamation Points!

So I have another *Face Palm* moment and think, “…Well, SQL Express 2008 R2 database size limit.  Ok, fun… now what…?”  Well Google here I come…

I found a few articles that talk about shrinking the database and truncating records…  It’s now 9:30 at night, I’m not feeling the “truncate my database” vibe tonight, so I continue reading on.  Eventually, I come across an article about cleaning up performance Data…  Well now, we don’t truly review performance data!  So why not clean that out, save some space and go home.  It’s now 9:50 ish…

So I have the plan of attack, and of course immediately roadblock #1 comes up: no SQL Management Studio on the box, and I don’t have another SQL box readily available.  So I back to google to find and download “SQL Server Express 2008 R2 with tools”…  except there are not stinkin’ tools in this package!  Argh!  Howver, hopping over to my MSDN subscription, I download “SQL Server 2008 R2 Express with Tools X86” and lucky me, the tools are actually there.  It’s now 10:30 ish…

Now roadblock #2: I took over this network and environment and I don’t know the account and password to use.  So I try my domain admin account…  Nope, public access only.  Well, I’m not a DBA — and I don’t claim to be a SQL Expert — so Google, here I come again… 

One of the top results is a Microsoft article on how to get access.  So in following this article, I stopped the Vmware services and added the –m parameter in the service start-up.  I proceeded to log in with my domain account (NOTE: RUN MANAGEMENT STUDIO IN PRIVLIDGED MODE, a.k.a. “Run As Administrator”) and my new shiny tools.  Once logged in, I created my account and provided myself with sysadmin rights to the instance.

I then backed up the databases to the D: drive (separate the OS and Database), and then ran the Script I got from the above VMware Article.  It worked like a charm, and in under a minute I cleared ½ of my database and was up and running.  Now, back to the Citrix support call…  It’s now 10:45… 

The End….?

 

[/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]
2017-07-27T00:01:02+00:00 June 18th, 2014|Uncategorized|

About the Author:


Fatal error: Uncaught exception 'GuzzleHttp\Exception\ClientException' with message 'Client error: `POST https://dc.services.visualstudio.com/v2/track` resulted in a `400 Invalid instrumentation key` response: {"itemsReceived":1,"itemsAccepted":0,"errors":[{"index":0,"statusCode":400,"message":"Invalid instrumentation key"}]} ' in /home/coretek/public_html/wp-content/plugins/application-insights/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 Stack trace: #0 /home/coretek/public_html/wp-content/plugins/application-insights/vendor/guzzlehttp/guzzle/src/Middleware.php(66): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response)) #1 /home/coretek/public_html/wp-content/plugins/application-insights/vendor/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response)) #2 /home/coretek/public_html/wp-content/plugins/application-insights/vendor/guzzlehttp/promises/src/Promise.php(156): GuzzleHttp\Promi in /home/coretek/public_html/wp-content/plugins/application-insights/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 113