Articles, Blogs, Videos, Podcasts, Share projects - Experiences from the field

Hi there,

After performing a System Clone, Patch or Upgrade, do you perform checks on your Instance? Checks to see how the state of the Instance is. And what kind of checks exactly? And are you performing these checks manually? And what if there are findings on these checks, are the findings actually System Clone, Patch or Upgrade related, or where the findings already pre System Clone, Patch or Upgrade applicable?

Pre- and post System Clone, Patch or Upgrade checks: Use Instance Scan

At recent customers I had a closer look at their Upgrade process (including System Clone) and simplified the whole process to 5 days. Using only out-of-the-box mechanisms like Upgrade Center, Clone Definitions, Automated Test Framework and Instance Scan. With Instance Scan the pre- and post System Clone and Upgrade checks were documented and (semi-)automated. Increasing the maturity level of the process, spotting issues earlier (or already pre System Clone or Upgrade), higher consistency in checks being performed, less manual errors, less time consuming, etcetera.

And not only to use post System Clone and Upgrade, though also to use pre System Clone and Upgrade! How often do issues appear... that actually have nothing to do with the System Clone or Upgrade. Issues that were already on the Instance before performing a System Clone or Upgrade. Sure, you still need to handle those issues For example queues not being processed, or having hundreds of restricted caller access privilege requests open. It helps massively knowing this beforehand, that it has nothing to do with a System Clone or Upgrade performed. This can prevent unnecessary discussions and red flags from being raised if you know such beforehand.

I now packaged similar generic checks which every company could use, added documentation as much as possible, and increased the checks to a total number of 88! Sounds like a lot for checks on your Instance, though I'm convinced you all can come up with even more checks that could or even should be performed pre-/post System Clone, Patch, or Upgrade. Just let me know in the comments.

Bonus: the same checks actually can be used as a recurring System Administrator activity.

Some of the checks you could argue about. Couldn't it be done within one check, instead of two or three separate ones? Or instead of applying a Scan Check for some, you could also apply a Notification or some other form of monitoring. It's also a bit of a personal preference I guess.

Concerning recurring System Administrator activities, you could definitely add a lot more checks! For example data related checks. Because this article is primarily focused on System Clone, Patch and Upgrade, I did not include data related checks (or only more important ones, like ones concerning the default admin account).

Scan Checks

I'll won't dive into the technical details of the Scan Checks created with Instance Scan, you can download the XML from Share, though below a list of the checks. You might recognize a few Scan Checks which appear to be also in the Instance Troubleshooter plugin from ServiceNow. I believe the checks I'm sharing are more complete. Using corrector techniques, for example like described in the articles which I wrote on performing Scan Checks only on production/sub production instance and performing Scan Checks only if certain plugins are active. Also some of the out-of-the-box Scan Checks simply contain mistakes, some producing incorrect results, and one even causing the Scan Check not to run at all(...).

Instance Scan - Sanity Suite

The XML from Share contains an Update Set with Scan Suite "Sanity" and the following 88 Scan Checks:

All email is sent to one email addressbm.scheduler account is inactvebm.scheduler account is locked outbm.scheduler account missingChecked out catalog item in productionChecked out workflows in productionCompleted Update Sets should be set to ignoreCustomer Update in progress in multiple Update SetsDebug properties should be disabled Default admin account admin role missingDefault admin account enable multifactor authentication checkedDefault admin account is inactiveDefault admin account is locked outDefault admin account missingDefault admin account security_admin role missingDefault admin account web service access only checkedDisable email debug logging Email reader schedule does not existEmail reader schedule is not runningEmail receiving non-operationalEmail receiving should be enabledEmail sending non-operationalEmail sending should be enabledErrored Flow engine contextErrored Workflow contextFlow Engine Event Handler schedule does not existFlow Engine Event Handler schedule is not runningguest user account is inactiveguest user account is locked outguest user account missingHigh number of flows running for a single record High number of workflows running for a single recordIdP certificate has changed or expiredImport Set Deleter schedule does not existImport Set Deleter schedule is not runninginstance.sec.user account is inactiveinstance.sec.user account is locked outinstance.sec.user account missingIn Progress Update Sets in productionLDAP server URL not operationalLong running Import Sets MID server downMID server not validatedMID server user without mid_server roleMID server version not same as instance versionml.admin account is inactiveml.admin account is locked outml.admin account missingNo active user has security_admin roleNo active user has sn_hr_core.admin roleNo active user has sn_si.admin roleOut-of-the-box POP3/SMTP accounts don't match the instance nameParent All Nodes/Active Nodes without childsRemote instance connection could not be verifiedRequested Restricted Caller Access PrivilegesSend all email to a test email addresssharedservice.worker account is inactivesharedservice.worker account is locked outsharedservice.worker account missingsharedservice.worker account platform_ml_create role missingsharedservice.worker account platform_ml_read role missingsharedservice.worker account platform_ml_write role missingSMTP sender schedule does not existSMTP sender schedule is not runningsn_ua.downloader account admin role missingsn_ua.downloader account is inactivesn_ua.downloader account is locked outsn_ua.downloader account missingsoap.guest user account is inactivesoap.guest user account is locked outsoap.guest user account missingTable Cleaner schedule does not existTable Cleaner schedule is not runningTest suite execution should be disabledText indexes not startedUncommitted Update Sets in productionUnprocessed eventsUnprocessed Flow engine contextUnprocessed incoming emailUnprocessed outgoing emailUnprocessed queuesUnprocessed schedulesUnprocessed skipped updates Unpublished flow / action in productionUpdate Scope Id is different than Update Set Scope Idvirtual.agent user account is inactivevirtual.agent user account is locked out

virtual.agent user account missing