Serge Chegorian's System Center Blog

Serge Chegorian's System Center Blog

[SCCM]: Limitation of use of the Orchestration groups

October 15th, 2020

Starting SCCM build 2002 Microsoft have introduced a new feature called Orchestration groups. You can find information about Orchestration groups here: https://docs.microsoft.com/en-us/mem/configmgr/sum/deploy-use/orchestration-groups.

The principal of Orchestration groups work is: you arrange several machines in a cluster, set an order and during the maintenance window SCCM deploys software updates in a pre-set order. There is also a possibility to run pre- and post-execution PowerShell script on each member.

However what Microsoft have overlooked is that if you are using Windows Defender as your primary Antivirus solution and distribute updates using ADR, Orchestration group would also consider Windows Defender updates as Windows Updates and because Windows Defender updates ADR is set to override the maintenance window, the next Windows Defender update will trigger Orchestration group in the following fashion:

  1. Windows Defender update triggers Orchestration group on Node 1 or Set 1.
  2. Orchestration group executes pre-deployment script on Node 1 or Set 1.
  3. Windows Defender update is deployed.
  4. There is no maintenance window so security updates are not deployed.
  5. Orchestration group timeout has expired and the group goes to ‘Failed’ state.

This issue is repeatable and was reported to Microsoft.

[SCCM]: Possible SCCM console issue after SCCM CB Uplift

January 24th, 2020

During the scheduled SCCM CB uplift we have observed the following issue: after the uplift SCCM Administration console started to crash on very specific property pages, i.e. Windows Updates client settings or scheduling regular OS image file servicing. Before the crash the console was giving us .NET errors saying that specific properties of WMI objects were not found.

The specifics of our environment is that we have SMS Provider installed on two redundant Management Point role servers and no SMS provider on the site server. Turns out that during the uplift SCCM does not update the smsprov.mof file on standalone SMS Providers.

The workaround is as follows:

  1. Verify whether or not <SCCM install dir>\bin\x64\smsprov.mof file is the same on the primary site server and on every standalone SMS provider server
  2. If and most likely not, replace smsprov.mof file on standalone SMS provider servers with the newer smsprov.mof found on the Primary Site Server.
  3. Compile smsprov.mof on each SMS Provider server with the command mofcomp smsprov.mof.

[SCCM] SCCM upgrade to 1906 – very important client consideration

August 21st, 2019

When you are planning for SCCM uplift to the build 1906 you must note that starting version 1906 SCCM client requires SHA-2 code signing support. What does that mean for you? It means that if your managed environment still have Windows 6.x OS systems (Windows 7, Windows 2008 and Windows 2008 R2), these systems require SHA-2 code signing support enabled. But do you really need to raise a change request and update legacy systems which are sooner or later will be decommissioned? There is another option. As you know SCCM client has ‘forward’ compatibility with newer infrastructure. That means that even older clients will work with SCCM 1906 but with the limited functionality. Microsoft call that feature ‘Extended Interoperability’ (EI). The following clients are recommended EI clients:

  • 1902 (5.00.8790)
  • 1802 (5.00.8634)
  • 1606 (5.00.8412)

I have selected the version 1902 as EI client for my environment which still contains a number of Windows 2008 R2 systems. To get my EI client in SCCM console I have selected both 1902 and 1906 updates for download. I need 1902 only as a source of EIC. The client binaries can be found in EasySetupPayload\<Configuration Manager 1902 Package GUID>\SMSSETUP\CLIENT. You have to copy all these files to the separate location because after the uplift to version 1906 this location will disappear. Then you have to create a collection for all your Windows 6.x OS systems and exclude this collection from the upgrade: exclude-client Now you have to upgrade the SCCM client on Windows 6.x systems by any mean. The client ver. 1902 will be capable to perform the main functionalities: software deployments, software updates and hardware inventory. The question is: would you be able to deploy the EI Client using SCCM. The answer is: yes! Package your EI client. Add a one-line cmd file to your package:

CCMSETUP.EXE /noservice /IgnoreSkipUpgrade /skipprereq:silverlight.exe SMSSITECODE=<site code> /source:%~dp0

Keep in mind that all your legacy systems must be in excluded from the automated client upgrade. /IgnoreSkipUpgrade switch will override this setting. Another important thing to know is that if the client version is older than the infrastructure version, ccmsetup exit code will be 7 instead of 0. However I have noticed that you don’t have to capture this exit code because your SCCM Client will be shut down and reinstalled. Application deployment cycle will detect the presence of the upgraded client during the next poll.

[SCCM]: Multiple applications are in ‘Waiting to install…’ state

May 8th, 2019

One of the most annoying problems on the SCCM client is when several applications stuck in ‘Waiting for install…’ state for days. Sometimes the reason for that is that one application is not distributed to the Distribution Point and for some reasons in SCCM build 1710 and older it blocks entire queue and does not allow other applications to install even though they are downloaded.

Here are 3 simple steps which would allow to identify the problem.

1. Identify that at least one application cannot be downloaded. For that go to SCCM client cache folder. There should be outdated  .BDRTEMP folder:

image1

2. Go to ContentTransferManager.log Look for the suspended job. Trace the log to find the corresponding ContentID

image5

3. GoTo AppDiscovery.log and identify the application name by ContentID

image3

Check if that application was correctly distributed.

[EUC]: It is time to move to UEFI boot

January 4th, 2018

If you are rolling out Windows 10 using SCCM infrastructure you may experience the following problems:

1. Newly build Operating System prompts for BitLocker key without changes to BIOS or hardware.

2. If you PXE boot using Legacy BIOS and then change BIOS settings to UEFI secure boot using vendor’s utility after reboot your system won’t be able to find any booting device.

The root cause of this issue is that starting Kapy Lake build (Intel 7th generation processor) CPU platform no longer support TPM 2.0/MBR boot combination.

The workaround is to change PXE boot to UEFI and set system BIOS to UEFI secure boot. Note that the factory settings are already UEFI secure boot.

This issue was observed on the latest models made by Dell and HP.

[SCCM]: SCCM Update stucks in ‘Downloading…’ state

December 5th, 2017

Sometimes when you try to download a new SCCM update in Updates and Servicing node it may stuck indefinitely in “Downloading” state. When you check dmpdownloader.log you may see the following error:

ERROR: Failed to download redist for 0f11caa4-7f7f-454b-96d6-75f427d015ce with command /RedistUrl http://go.microsoft.com/fwlink/?LinkID=857597 /LnManifestUrl http://go.microsoft.com/fwlink/?LinkID=854716 /RedistVersion 201706 /ProxyUri http://companyproxy:81/ /ProxyUserName CONTOSO\SCCM_Admin /ProxyUserPassword 3082018F06092A864886F70D010703A08201803082017C020102318201303082012C0201028014A8 /NoUI  “\\SCCM01.contoso.com\EasySetupPayload\0f11caa4-7f7f-454b-96d6-75f427d015ce\redist”

This error is not very descriptive however it indicates the problem with SCCM prerequisites download. Additionally when you go to Program Files\Microsoft Configuration Manager\EasySetupPayload folder you would see both 0f11caa4-7f7f-454b-96d6-75f427d015ce folder and 0f11caa4-7f7f-454b-96d6-75f427d015ce.cab file. The code may be different depending on the upgrade.

The root cause of this issue is your firewall or proxy or something else which blocks specific files from downloading assuming this is a security threat. The workaround is:

Go to 0f11caa4-7f7f-454b-96d6-75f427d015ce\SMSSETUP\BIN\X64 ad copy SetupDL.exe to the computer with has unprotected access to Internet. Run SetupDL.exe <Destination folder>. Once download is completed, copy all files and folders structure to 0f11caa4-7f7f-454b-96d6-75f427d015ce\redist folder and run updates check or (better) restart SMS_EXECUTIVE service. SCCM will detect your files and 0f11caa4-7f7f-454b-96d6-75f427d015ce.cab file should disappear.

If you do not have “clean Internet” running SetupDL.exe output would at least give you an indication what file cannot be downloaded. In that case download ConfigMgr.Manifest.cab and ConfigMgr.LN.Manifest.cab using links from the error message above (/RedistUrl http://go.microsoft.com/fwlink/?LinkID=857597 and  /LnManifestUrl http://go.microsoft.com/fwlink/?LinkID=854716), unpack both cab files, open .xml manifest files in Notepad and download every missing prerequisite file individually using HTTP links from the manifest files.

[SCCM]: SCCM Upgrade Path from Branch 1602 to Branch 1706 or above

December 5th, 2017

Just a quick note. You may miss several SCCM branches and are still on version 1602 or 1606 of SCCM. Despite stated by Microsoft direct upgrade from Branch 1602 to 1706 is not possible. The maximum version you can upgrade from Branch 1602 is 1702 (even though you can see versions 1706 and above in the manifest file). So the actual upgrade would be three steps process: from 1602 to 1702, download 1706 and from 1702 to 1706. Same applies to the branch 1606.

Keep this in mind when planning for SCCM upgrade.

[SCCM 2012 R2]: multiple SMS__SMS_SQL_SERVERXXX folders are created on remote SCCM SQL

April 27th, 2017

Sometimes when you have a dedicated SCCM Database server or SQL cluster you may notice that SMS_<FQDN>_SMS_SQL_SERVERXXX folder is created on the C: drive of the SQL server or a cluster every 3 minutes where FQDN is the name of your SCCM site server. This happens because SCCM Site Component Manager is not flagged that SMS_SITE_SQL_BACKUP_<FQDN> service is installed so SMS_SERVER_BOOTSTRAP_<FQDN>_SMS_SQL_SERVER creates setup folder for SMS_SITE_BACKUP… over and over again

Solution

Go to the site server SMS_<Site Code>\inboxes\certmgr.box and check for out-dated CMN files. Delete them.

[SCCM 2012 R2]: Troubleshooting database replications and service broker issue

December 24th, 2016

Last week I dealt with a very interesting and unusual SCCM failure. It has started with a link failure error between CAS and one of the primary sites. When I ran Replication Link Analyser the first error message was “SQL Server Broker login is missing for sites: <my primary site code>”. After that RLA informed me that the login is recreated but in fact it was not and the issue was still there. I was also unable to find any information on how to recreate SQL Server Broker or at least what it is.

After more rigorous search I have found the following SQL command which shows you SQL replication status in real time.

Use CM_CAS; Select * from sys.transmission_queue

The content of that table should change dynamically. In my case there was a bunch of stalled messages with ConfigMgr_Site<My Primary Site Code (PSC)> in to_service_name column and “Connection attempt failed with error: 10060” in transmission_status column. That gave me clear indication that the Service Broker transmission is broken between my CAS and PSS.

Note: when the transmission is resumed SQL should clear up stuck messages however sometimes you might need to clear them up yourself using update sys.transmission_queue Also please note that any intervention to the SCCM database is not supported by Microsoft.

After several telnet tests I have figured out that Service Broker is not responding or listening on PSS server database.

In our environment all SQL servers are shared hosts so all Service Brokers are using private ports. To identify the port used by Service Broker run the following SQL script on your SQL instance

Use CM_CAS select port from tcp_endpoints where type_desc like ‘%SERVICE_BROKER%’

Note that there could be only one Service Broker endpoint per database.

I have executed the query above and the result was nil. That gave me an understanding that somehow my Service Broker was deleted on the database.

At that stage I was about to give up. There is a script which creates Service Broker endpoint but I know that SCCM secures all internal communications with certificates had no idea which certificate to use. I’ve been thinking to either call Microsoft or reinstall the site (including several role servers) but fortunately I have found the required script on Internet.

CREATE ENDPOINT [ConfigMgrEndpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = <my port>, LISTENER_IP = ALL)
FOR SERVICE_BROKER (MESSAGE_FORWARDING = ENABLED
,
MESSAGE_FORWARD_SIZE = 5
, AUTHENTICATION = CERTIFICATE
[ConfigMgrEndpointCert]
, ENCRYPTION = REQUIRED ALGORITHM AES)
GO

All good but how would I know what private port was used by my missing Service Broker? In SQL Management Studio go to CM_CAS\Service Broker\Routes\ConfigMgrDRSSiteRoute_<your PSC>, click on properties and in properties check for Address which would look like TCP://<your PSS FQDN>:<port>.

Once I’ve executed the SQL script above my telnet test has succeeded. I ran RLA again and it gave me “SQL Server Broker login is missing for sites: <my primary site code>” error again but this time it’s succeeded in fixing this issue and this error has not re-appear again.

I thought now it’s just a matter of time but in several hours I still saw no activity in rcmctrl.log. However all error messages from sys.transmission_queue have gone.

So I had another look at the link status, specifically at Initialization Detail tab. It is very important to look at it from both sides, i.e. both CAS and PSS. On PSS side I’ve noticed that one replication group has stuck at 1% replicating up to CAS.

There is a way to reset replication group. You have to create <replication group name>.pub file and place it to the rcm.box inbox. This file should disappear in 5-10 seconds. If it does not disappear at all it clearly indicates that the issue is on another end. Delete it and try from another side.

Once I’ve dropped the PUB file to the rcm.box it has pulled the plug. I have started to see replication activity in rcmctrl.log and file exchange in rcm.box. The issue has gone in hour and a half.

Several important things to remember when you have SCCM 2012 replication issue:

  • Start your troubleshooting with RLA.
  • If the primary site sits in link failure state for substantial amount of time SCCM puts the primary site in read only mode and the link in the maintenance state
  • If the issue is not fixed SCCM will also put CAS database in maintenance mode, consequently the rest of links will fail.
  • Check sys.transmission_queue for stuck transactions. The content of this table must rapidly change.
  • Check rcmctrl.log for any activity.
  • Identify your Service Broker ports and run telnet connectivity tests.
  • Check CM_<site code>\Service Broker\Queues if any queue is down.
  • The easiest way to reset replication group replication is to drop <replication group name>.pub file to the rcm.box inbox. Note that the PUB file name should be <replication group name>.pub on PSS, on CAS it should be <replication group name>-<primary site code>.pub. The file should disappear in 5-10 seconds. If it does not, the issue is on another end. Delete the file and try on another end.

[SCCM 2012 R2]: SUP becomes out of synch

October 7th, 2016

Sometimes SMS_WSUS_SYNC_MANAGER may report error Message ID 6703:

WSUS Synchronization failed. Message: timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

At the same time the rest of infrastructure is healthy. According to wsyncmgr.log an attempt to synchronize only lasts for 1-2 minutes. SUP reinstall does not help.

Go to WSUS manager\Options\Update Source and Proxy server. You may see the notification:

Cannot save the configuration because the server is still processing a previous configuration change.

This is a culprit! In order to get rid of it you may uninstall SUP, reinstall WSUS (including database) and install SUP. There is an easier way to fix it.

Start SQL Server Management Studio, connect to WSUS database and execute the query:

UPDATE tbSingletonData
SET ResetStateMachineNeeded = 0

Restart Windows Update service. Check to confirm that the warning has gone. The next scheduled synchronization should be successful.

SCCM Keeps Processing Package

December 4th, 2015

Sometimes you may see several hundred thousand informational messages produced by the child site distribution manager. The messages look like this:

SMS Distribution Manager successfully processed package “Java 7 Update 91” (package ID = CAS000D3).
SMS Distribution Manager is beginning to process package “Java 7 Update 91” (package ID = CAS000D3).
SMS Distribution Manager successfully processed package “Java 7 Update 91” (package ID = CAS000D3).
SMS Distribution Manager is beginning to process package “Java 7 Update 91” (package ID = CAS000D3).
SMS Distribution Manager successfully processed package “Java 7 Update 91” (package ID = CAS000D3).
SMS Distribution Manager is beginning to process package “Java 7 Update 91” (package ID = CAS000D3).

This package looping may affect both existing and non-existing packages.

Workaround:

On the affected server go to inboxes\distrmgr.box folder.

Select and delete CAS000D3.PKG and CAS000D3.PKN files

Connect to CAS DB and run the following query:

SELECT * FROM PkgServers where NALPath like ‘%<affected server name>%’ and PkgID = ‘CAS000D3′

If this query returns any result redistribute CAS000D3 package.

 

SCOM 2012: Database dropdown menu is blank when installing a new management server

October 16th, 2015

When installing and specifically reinstating a new management server, setup program is able to establish connection to the operational database server but the dropdown menu is blank (not disabled or greyed out).

Most likely this is because your server already exists in SCOM database.

If the server is former MS delete it from the SCOM console and start the installation process from the beginning.

SCCM: How to delete all packages from the DP scheduled for decommissioning

September 4th, 2015

Prior to the decommissioning of the Distribution Point it is recommended to delete all packages assigned to this DP. If you do not do that the ‘orphaned’ packages may sometimes appear on your software distribution reports producing unnecessary noise.

This simple PowerShell one liner removes all packages from the selected DP. Note that this script does not physically remove the package from DP, it simply deletes the relevant record in the database.

Let’s assume CEN is your Central or standalone SCCM site. This is the script:

gwmi -Namespace root\sms\Site_CEN -query "select * from SMS_DistributionPoint where ServerNALPath like '%MYDP001%'" | % ($_.Delete())

Sometimes this script may give you an error. This may happen because CEN is your central site and the package is published on one of the primary sites. To identify package’s source site simply modify the script:

gwmi -Namespace root\sms\Site_CEN -query "select * from SMS_DistributionPoint where ServerNALPath like '%MYDP001%'" | % ($_.SourceSite)

And then change CEN site code in the namespace root\sms\Site_CEN to the site code you have found.

SCCM 2012 Collection Evaluation: Maximum recursion 100 has been exhausted

August 11th, 2015

Sometimes you may see the following error message in SCCM colleval.log file:

Maximum recursion 100 has been exhausted

This typically happens if for one of the collections the collection’s Limit to collection parameter is set to itself. Also if this has happened you might not be able to manage, update or delete the collection in trouble.

In order to fix it you have to go the site database via SQL Management Studio and execute the following command:

UPDATE Collections_G SET LimitToCollection =’XYZ00001′ WHERE SiteID=’XYZ00056′

where XYZ00056 is your collection in trouble and XYZ00001 is any valid collection.

Issue with editing SCOM 2012 Subscription criteria via GUI

August 7th, 2015

When editing SCOM 2012 Subscription criteria via GUI you may have the following error message:

“The criteria associated with this notification subscription are of a form not supported by the Operations Manager Console. You may continue through this wizard, and the criteria will remain unchanged.”

That typically means that your criteria are referencing to the non-existing rule or monitor. This is how to fix it:

Go to SCOM PowerShell. Get your criteria using the following command:

Get-SCOMNotificationSubscription | Where-Object {$_.DisplayName -eq “<Your faulty subscription name>”} | Format-List DisplayName,@{Label=”Criteria”,Expression={$_.Configuration.criteria}}

This will produce XML file with the following format:

<Expression>
<SimpleExpression>
 <ValueExpression>
  <Property>ProblemID</Property>
 </ValueExpression>
 <Operator>Equal</Operator>
 <ValueExpression>
  <Value>aaa0000-0000-0000-0000-000000000000</Value>
 </ValueExpression>
 </SimpleExpression>
</Expression>

Every value represents a rule or monitor ID. Extract all these values. For each ID run the following command:

Get-SCOMMonitor -id aaa0000-0000-0000-0000-000000000000 | Select-Object Name

This will give you the rule or monitor name. The ones returning no result are your non-existing ones. Record names for the existing rules or monitors.

Delete the faulty subscription and recreate it from scratch. Now you have all your criteria.

SCCM2012: IP Subnet vs. IP Range Boundary

June 1st, 2015

I was unable to find any reference on Microsoft network but eventually this is confirmed after a number of tests. No matter what you put as a subnet mask whent creating a new IP subnet site boundary, but SCCM is always using 255.255.255.0 (/24) mask.

So if you want to have a wider range of addresses (in my case it was /20) use IP Range boundary instead of IP Subnet boundary.

How to prestage a large number of applications in SCCM 2012

May 29th, 2015

In some circumstances specifically if you are dealing with a slow network package distribution over WAN may be unreliable and it would worth to prestage the content and then copy it to the remote Distribution Point using an alternative method, for example media sent with a courier. So the question is how to script out prestaging process for a large number of applications.

In SCCM 2012 SP1 onwards there is a new cmdlet Publish-CMPrestageContent. The syntax of this command is:

 

Publish-CMPrestageContent -<ApplicationId|PackageID|DriverPackageID|OperatingSystemImageID> <String[]>  -DistributionPointName <String[]> -FileName <String[]>

Where:

DistributionPointName is FQDN of the Distribution Point which already has a copy of the package. You cannot prestage a package until it is successfully distributed to at least one DP.

FileName is a prestage file name

For example:

Publish-CMPrestageContent -PackageID “XYZ000C0″ -DistributionPointName “XYZSERVER.LOCAL” -FileName “C:\Temp\XYZ000C0.pkgx”

The tricky bit here is that ApplicationID is a new, SCCM 2012 specific package ID which is different to the “classic” site code plus 5 digits package ID format. Fortunately every application has both new and classic ID’s. But keep in mind that if you decide to use SMS_Application WMI class in order to identify ApplicationID by PackageID it would fail because PackageID property of SMS_Application class is so called “lazy” property and is not populated.

You have to use SMS_CIContentPackage class or v_CIContentPackage view if you prefer SQL in order to establish relation between PackageID and ApplicationID.

In order to perform a bulk import of the prestaged content on the remote Distribution Point put all PKGX files in the same folder and then run this one line command:

for /r c:\temp %s in (*.pkgx) do extractcontent /P:%s /S

A couple of important notes at the end:

  1. By default SCCM Powershell execution policy is set to AllSigned and cannot be reset to Unrestricted using Set-ExecutionPolicy. If you run a script set it to RemoteSigned.
  2. SCCM Powershell does not execute scripts outside SCCM environment. If you change SCCM Powershell to any local drive your script would not execute.

Co-hosting SCCM PXE point, DHCP and WDS services

January 22nd, 2015

SCCM PXE point or SCCM PXE enabled distribution point can be co-hosted with DHCP services. This configuration though is not recommended but is fully supported by Microsoft.

In order to configure DHCP and Windows Deployment Services on the same host you must do the following:

1. Install DHCP and WDS

2. For DHCP configure option 60 with with the value PXEClient. Do not enable and configure options 66 and 67.

3. Go to WDS properties, tick an option ‘Do not listen on port 67

Now you can start using WDS or configure SCCM PXE option on the top of WDS

SCCM 2012 Software Updates do not install

August 25th, 2014

In SCCM 2012 you may come across the situation when you crate a Software Update package but it does not install. You may check DP, policy, client and you see it is detected but is still not deployed.

Have a look at your package size specifically the number of updates. Exactly like for application each Software Update has it’s maximum runtime which is unmanageable in SCCM 2007 and can be changed in SCCM 2012 R2 (10 minutes by default). The total runtime for the package is a sum of maximum runtimes for every individual Software Update. If this sum exceeds 24 hours your package will never run because SCCM believes there is no window to run it.

Split up your package in several smaller packages and it will work.

SCCM 2007 R3 Reporting Point Error 500

August 1st, 2014

Sometimes when you access SCCM 2007 reports running on Reporting Point you may see HTTP Error 500 – Internal server error. If it affects some of your reports where you would anticipate a large number of records in the output it could be caused by insufficient buffer size. This is a well known issue and the work around it can be found here.

But what if all your reports are affected? Try several reports again and go to IIS logs. You will see something like this:

GET /SMSReporting_XXX/Report.asp ReportId=200|372|ASP_0177_:_8007007e|Server.CreateObject_Failed 80

This means that one of Reporting Point ActiveX is missing.

Check <webroot>\SMSComponent folder. If consistent it must contain the following files:

    FormatMessageCtl.dll
    smscomponent.dll
    SMSRPH.exe

Most likely FormatMessageCtl.dll will be missing

Go to SCCM 2007 SP2 set up DVD, SMSSETUP\BIN\I386

Run reportinginstall.exe /x

Select the file FormatMessageCtl.dll (you will see 3 instances but they are all the same) and extract it to <webroot>\SMSComponent folder.

That’s it, you don’t need to register this DLL or restart IIS and SCCM. Just run report again.


Why did it happen?

Go to <wwwroot>\SMSReporting_<site code> folder and check for install.log file. Check the file time-stamp and dates in the file. Most likely your Reporting Point was recently reinstalled.

SCCM Component Manager service “pings” all installed components every 3600 seconds. If it does not have a response for several consequent attempts it reinstalls the component. This is the feature of SCCM 2007 and according to Microsoft Premium Support it cannot be configured or adjusted. So apparently something has happened to your Reporting Point which has triggered component reinstallation.

Serge Chegorian's System Center Blog

Serge Chegorian's System Center Blog