Certificate Services Did Not Start on a Sub CA…

2017-07-27T00:01:02+00:00 June 26th, 2014|Uncategorized|

Hi Internet friends!  I recently came across a two-tier PKI infrastructure (Enterprise-style, with offline root) that was set up for a customer who was transitioning to a brand new AD Forest.  The customer stated the CAs were working just fine when initially built, and had the validation documentation to prove it.

However, a couple months later, the customer was receiving errors when trying to generate a certificate from the subordinate CA for a new application infrastructure, and they asked for help.  They saw this error in Event Viewer:

Event ID 100: Active Directory Certificate Services did not start: Could not load or verify the current CA certificate.  <subCA certificate name>  The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).


Odd that it worked at implementation, but suddenly stopped working.  It was time to check things out and review the implementation.

The first thing I noticed was that the Active Directory Certificate Services service (certsvc) would not start.  When I tried it from the Certification Authority GUI I saw the error message shown at left.

You’ll note that the GUI message is nearly identical to the Event ID 100 message.  I simply mention this so if you do see the message in the GUI, you don’t need to run to Event Viewer; you’ve got the same issue I experienced at the customer site.

Web searching was only slightly helpful and put me in the ballpark for the fix, but it didn’t bring me all the way home.  And if you simply want my quick steps on how I corrected the issue, here is what I did:

  1. Powered up the offline root CA
  2. Renewed the Subordinate CA certificate with the Root CA and re-installed it on the Subordinate CA
  3. Regenerated the Root CA CRL and copied it to the correct location on the Subordinate CA
  4. Started the AD CS service on CA1, the Subordinate CA.


But if you want more detail on what I discovered, here’s the meat…

First off, the Root CA CRL was set to expire in just seven days.  That’s not going to be good.  That right there prevents the certsvc service from starting after the Root CA Certificate Revocation List (CRL) expires.  The service depends on finding Root CA CRL.  If it is expired – or can’t find it – the service will not start.

Simply regenerating that and replacing it on the Subordinate CA – after adjusting the CRL Publication interval to 1 Year – was the first thing I tried, but it didn’t help.  This led me to investigate if the Subordinate CA could even find the Root CA CRL. 

I opened the Subordinate CA certificate, clicked the Details tab, and checked the value in the CRL Distribution Points entry, shown here at right.

That’s where the light bulbs started to get bright.  The server was named ca1.domain.com, but they had their PKI IIS set up with the URL pki.domain.com/pki.


I checked the CRL Distribution Point settings on the Root CA and the Subordinate CA.  Ah ha!  They were not matching what was in the Subordinate CA cert.

Now take a look at the Root CA CRL CDP Settings in the Properties page at left.  You’ll see that this is set to the URL I mentioned before, pki.domain.com/pki – which is *not* what is in the Subordinate CA CRL Distribution Points property of the certificate!

The fix was now pretty apparent.  All I needed to do was renew the Subordinate CA certificate and replace it on CA1.

Take a look below at the CRL Distribution Point property on the new Subordinate CA cert and you’ll see it now matches what is in the Root CA settings.


From there it was just adjusting the Root CA CRL publication interval in the Certification Authority MMC by right clicking Revoked Certificates, selecting Properties, then setting it according to Microsoft Best Practices of 1 Year, as shown here.

Then, I right clicked on Revoked Certificates, selected All Tasks, Publish, and re-published the CRL and copied it to the /pki folder on the Subordinate CA Server.

Viola!  The services now started on CA1, the Subordinate CA, and everything worked as expected.

Now it’s true that this does mean that the Root CA must be powered up annually so the CRL can be regenerated and copied, but there is no getting around that. 

Yes, of course you can set the CRL publication interval to be longer than a year; but as per best practices, you should set that to expire in the least amount of time that is tolerable.  And in this customer’s case, the recommendation of each year was suitable and appropriate.


Firing up the Root CA and doing this annually isn’t a big deal in my world, hence that recommendation.

In the end, with the certificate service running and the CRLs fixed, the customer was able to mint the certificates necessary for the new application, and I made an appointment with the customer in my calendar for just shy of a year from now…

I hope that helps someone else out there!

The Red Exclamation Points (A Suspense/Thriller Mini-Novella)…

2017-07-27T00:01:02+00:00 June 18th, 2014|Uncategorized|

This evening was going like any other busy evening for a Coretek employee with a deadline fast approaching.  I was working with Citrix Support to figure out what the problem is with a Citrix XenMobile (an Enterprise Mobile Device Manager), when the dreaded warning texts started pouring in…

I received message after message about “servers down, 8:30 PM”…  The nice thing about it is I figured out who was by their phones to respond to the alerts.  After a quick couple of text messages, I acknowledged the alerts and began troubleshooting. 

I knew the issue was with our VMWare Vcenter Server, and my first reaction was, “…well I’m busy let’s just reboot this thing.”  So I rebooted the server and waited for it to idle down before launching VSphere, which now showed a bunch of orphaned VMs (“orphaned” is a bad word in Vmware, essentially meaning that a portion of the vm be it disk or configuration files can’t be located). 

So my mind starts racing to a network or storage issue.  I go to click on an orphaned VM and vSphere Crashed.  I now *Face Palm* myself (…sigh…) and go back to the vCenter server and attempt to start the service “VMware VirtualCenter Server.”  It starts, and as I open vSphere again, and it almost immediately crashes. 

So now I’m starting to get really nervous, and I’m thinking I’m going to need to do a restore; meanwhile, the support engineer from Citrix is on the line asking me to show her my Netscaler configuration and AppController configs.  I tell her, “I’ll show you what I have in a minute, ½ of my network just alerted me it is down.”  I believe it was one of those “lost in translation” moments because she waits and a minute later says, “well can I see the configs?”.  I give in show her the configs and run for another laptop from my desk to continue troubleshooting.  And as I log in and open Event Viewer on the vCenter server to see what is going on…  All I see are red exclamation points…


Red Exclamation Points!

So I have another *Face Palm* moment and think, “…Well, SQL Express 2008 R2 database size limit.  Ok, fun… now what…?”  Well Google here I come…

I found a few articles that talk about shrinking the database and truncating records…  It’s now 9:30 at night, I’m not feeling the “truncate my database” vibe tonight, so I continue reading on.  Eventually, I come across an article about cleaning up performance Data…  Well now, we don’t truly review performance data!  So why not clean that out, save some space and go home.  It’s now 9:50 ish…

So I have the plan of attack, and of course immediately roadblock #1 comes up: no SQL Management Studio on the box, and I don’t have another SQL box readily available.  So I back to google to find and download “SQL Server Express 2008 R2 with tools”…  except there are not stinkin’ tools in this package!  Argh!  Howver, hopping over to my MSDN subscription, I download “SQL Server 2008 R2 Express with Tools X86” and lucky me, the tools are actually there.  It’s now 10:30 ish…

Now roadblock #2: I took over this network and environment and I don’t know the account and password to use.  So I try my domain admin account…  Nope, public access only.  Well, I’m not a DBA — and I don’t claim to be a SQL Expert — so Google, here I come again… 

One of the top results is a Microsoft article on how to get access.  So in following this article, I stopped the Vmware services and added the –m parameter in the service start-up.  I proceeded to log in with my domain account (NOTE: RUN MANAGEMENT STUDIO IN PRIVLIDGED MODE, a.k.a. “Run As Administrator”) and my new shiny tools.  Once logged in, I created my account and provided myself with sysadmin rights to the instance.

I then backed up the databases to the D: drive (separate the OS and Database), and then ran the Script I got from the above VMware Article.  It worked like a charm, and in under a minute I cleared ½ of my database and was up and running.  Now, back to the Citrix support call…  It’s now 10:45… 

The End….?



Repair Broken Hyper-V Integration Services…

2017-07-27T00:01:02+00:00 June 12th, 2014|Uncategorized|

Recently, I ran a bunch of Windows patches and upgraded the Hyper-V Integration Services at the same time on a Hyper-V Windows 7 guest in my lab. 

Error 1083

After a reboot, a few of the Hyper-V Integration Services would not start correctly.  Honestly, I wouldn’t have even noticed initially if I weren’t regularly running a PowerShell shutdown script that calls my VMs with a “Stop-Vm”, that was suddenly unable to gracefully shut down this newly-patched VM.  I’m not sure if the simultaneous patching and updating directly caused my error symptoms, but I’m guessing the update was unable to write the registry at the same time as some patch that was installing. 

In trying to restart the service manually, I got the following error:

Error 1083: The executable program that this service is configured to run in does not implement the service.



It turns out that I actually had three serivices failing (vmicshutdown, vmicvss, vmictimesync).  I did some searching, and fortunately a user named “gzzhouch” at the Windows Server Forum was following up an old post (thanks!) with a similar issue…  And I was able to follow his recommendation and get everything back in order. 

Let’s step through the fix of the Hyper-V Time Syncronization Service in a little more detail, which was one of the few that failed for me.  First, open the Services MMC, look at the details of the failing service, and note the “Service name” and the item at the end of the “Path to executable”, as in my example in the image at right.

…in my example case, the Service Name is “vmictimesync”, and the path item is “LocalServiceNetworkRestricted”.  Next, open regedit (don’t do this if you are not comfortable editing the registry!), and browse to:

HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionSvchost

Edit the Registry

…and open the value that matches your path item (in my case “LocalServiceNetworkRestricted”), and add the service name (in my case “vmictimesync”) to the list, like this:


…and do this for each Integration Service for which you receive the error. 

I’ll warn you though, be very thorough; it can trick you since LocalServiceNetworkRestricted and LocalSystemNetworkRestricted look almost the same (ask me how I know). 

Then reboot, and that should do it!  At least it did for me.  Enjoy!



2017-07-27T00:01:02+00:00 June 4th, 2014|News|

Farmington Hills, MI – June 4th, 2014 – Due to tremendous growth over the past year, Coretek has expanded its operations team and hired Bill Deighton as Chief Operating Officer.  In this new role, Bill will provide the vision, leadership and management necessary to ensure Coretek has the proper operational controls, as well as people and systems in place, to effectively grow the organization.  Under Bill’s leadership, Coretek will further enhance operational efficiencies to provide the highest level of consulting services, project management and technology solutions.

“Coretek Services is committed to creating high-value technology solutions for healthcare and enterprise clients across the country” stated Ron Lisch, Coretek Services’ chief executive officer.  “Bill has the experience and expertise needed to ensure operational excellence, financial strength and high-value products and services.”

Bill brings over 28 years of diverse experience in the healthcare, retail, distribution, and manufacturing industries.  He holds a bachelor’s degree in Finance from Eastern Michigan University, along with numerous technical and managerial certifications.  Bill’s broad professional experience spans roles in financial analysis, application development, business/systems analysis, supply-chain management, operations management, strategic planning, and HIPAA/HITECH Security Compliance Officer.  For the last 18 years, he has lead the Information Technology and Services departments for two multi-million dollar organizations as VP/Director of Information Technology.