Holistic Patching for Oracle Enterprise Manager

Simple and Secure: Holistic Patching for Enterprise Manager

This blog outlines how holistic patching works and provides an example of it in action. Holistic patching for Enterprise Manager is a newly announced(May 2024) method for applying all stack patch bundles for a given release by Oracle across all its components. Sumesh Balakrishnan from Product Management of EMCC has a blog post(Holistic Patching – a modern approach for Security Patch Management) about this new method and also How To Apply/Rollback Holistic Patch (SPB, CPU, and JDK) On OEM 13.5 (Doc ID 3002450.1) document explains all the steps. I will use them as a guide.

Patching Enterprise Manager Cloud Control is challenging because it involves 10-15 different components that require patching, such as OHS, WLS, OPSS, ADR, ADF, RDA, OWSM, JDK and OPatch. Some components require a maintenance window, and it takes hours to fully patch the enire system. In my environment, Oracle Enterprise Manager Cloud Control is effectively used, primarily for Fleet Patching.

Holistic patching simplifies patch management by consolidating updates into a single bundle for the quarterly cycle, reducing complexity and downtime. This efficiency allows organizations to focus more on core activities. When patching all the stack for EMCC, the biggest challenge is keeping the JDK up to date, as it often has many prerequisite patches. For offline environments, downloading all the setups and using different patching methods (OMSPatcher, OPatch, Java, opatch napply, unzip, etc.) can be particularly cumbersome.

With the new methodology, only the OMSPatcher will be updated to the latest version, and all subsequent patching will be done using a simple command. Consolidating patch activities into one downtime window minimizes disruption and reduces the overall apply time since System and environment prechecks are performed once.

OMSPatcher now includes new parameters such as -spb_patch and -jdk_update. We will use the -spb_patch parameter to patch the entire stack. However, for OMS on AIX, the -jdk_update parameter must be explicitly included in the omspatcher apply command. For other Unix platforms (LinuxX64, Solaris, etc.), the -jdk_update parameter is not needed, as the JDK will be updated automatically as part of the apply command.

According to the Readme of Patch 36635982, the patches included in this Holistic Patch are provided below. It will be awesome to patch all at once. It is a big bundle stack patch that includes(OHS,OPSS,WLS etc.) all except database.

• Bug 36348444 ADF BUNDLE PATCH 12.2.1.4.240228
• Bug 34809489 PS4 : PATCH CONFLICT OBSERVED BETWEEN 34647149 AND 34604561
• Bug 35965629 – ADR FOR WEBLOGIC SERVER 12.2.1.4.0 – SIZE OPTIMIZED FOR JAN 2024
• Bug 36410345 Coherence 12.2.1.4 Cumulative Patch 21 (12.2.1.4.21)
• Bug 33093748 FMW PLATFORM 12.2.1.4.0 SPU FOR APRCPU2021
• Bug 36468190 FMW Thirdparty Bundle Patch 12.2.1.4.240401
• Bug 32720458 Fix for Bug 32720458
• Bug 35430934 MERGE REQUEST ON TOP OF 12.2.1.4.0 FOR BUGS 32720458 33607709
• Bug 34153238 HTTPS PROXY CONFIGURATION IS NOT USED WHEN PROTOCOL IS CONFIGURED TO TCP
• Bug 31657681 THREADS CONTEND FOR LOCK IN LOADFILEBASEDKEYSTORE WHEN OPENING TLS/SSL ENABLED JDBC CONNECTIONS
• Bug 36270037 Oracle JDK 8 Update 411
• Bug 28186730 OPATCH 13.9.4.2.15 FOR EM 13.5 AND FMW/WLS 12.2.1.4.0 and 14.1.1.0.0
• Bug 36316422 OPSS Bundle Patch 12.2.1.4.240220
• Bug 35024228 OSS BUNDLE PATCH 12.2.1.4.230128
• Bug 34065178 MERGE REQUEST ON TOP OF 12.2.1.4.0 FOR BUGS 34010500 33903365
• Bug 36402397 OWSM BUNDLE PATCH 12.2.1.4.240313
• Bug 34831198 PERL PATCH FOR EM 13.5 FOR CVE-2022-43680: LIBEXPAT UPDATE TO 2.5.0
• Bug 36187017 RDA release 24.2-2024416 for FMW 12.2.1.4.0
• Bug 36349529 WebCenter Core Bundle Patch 12.2.1.4.240227
• Bug 36440005 WLS PATCH SET UPDATE 12.2.1.4.240325
• Bug 36636070 EM Stack Patch Bundle 13.5.0.0.0(ID:240520.101106) (Patch 36635982)

I will apply 36335368 Holistic Patch. Its prerequisites are OMS RU 21 or later on OMS_HOME. We have OMS RU 19 so we will apply release update on OMS.

• OMS RU 21 (36172321) also requires one-off Patch 36492660.
• If you choose to apply RU 22 (36335368), it also requires one-off patch 36634909.

First I will apply 36172321 OMS RU 22. It also has prerequisites and I have all in my inventory.

    [oracle@blt01 oms]$ OPatch/opatch lsinventory | grep 35430934
    Patch  35430934     : applied on Mon Feb 12 14:56:07 EET 2024

    [oracle@blt01 oms]$ OPatch/opatch lsinventory | grep 34153238
    Patch  34153238     : applied on Mon Feb 12 14:52:50 EET 2024

    [oracle@blt01 oms]$ OPatch/opatch lsinventory | grep 31657681
    Patch  31657681     : applied on Mon Feb 12 14:50:41 EET 2024

We have upgraded current ompspatcher version to the latest.

[oracle@blt01 ~]$ cd /u01/app/Setup/EMCC_Setups/
[oracle@blt01 Setup]$ unzip p19999993_135000_Generic.zip
[oracle@blt01 patches]$ mv  /u01/app/oracle/em135/oms/OMSPatcher /u01/app/oracle/em135/oms/OMSPatcher_old
[oracle@blt01 oms]$ cp -r /u01/app/Setup/OMSPatcher /u01/app/oracle/em135/oms/

To avoid setting credential every time we will use the property file we have created before. It can be created with commands provided below.

[oracle@blt01 oracle]$ mkdir -p /u01/app/oracle/property_file
[oracle@blt01 oracle]$ cd /u01/app/oracle/em135/oms/
[oracle@blt01 oms]$ OMSPatcher/wlskeys/createkeys.sh -oh /u01/app/oracle/em135/oms -location /u01/app/oracle/property_file

This command creates config and key files. We provide them to the variables shown below.

[oracle@blt01 property_file]$ vi prop_file
AdminServerURL=t3s://blt01.localdomain:7102
AdminConfigFile=/u01/app/oracle/property_file/config
AdminKeyFile=/u01/app/oracle/property_file/key

Lets apply Release Update on OMS.

[oracle@blt01 Setup]$ emctl stop oms		
[oracle@blt01 Setup]$ unzip p36335368_135000_Generic.zip
[oracle@blt01 Setup]$ cd 36335368/
[oracle@blt01 36335368]$ export PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin:/u01/app/oracle/product/19.3/dbhome_1/bin:/u01/app/oracle/em135/oms/bin:/u01/app/oracle/em135/oms/OMSPatcher	

[oracle@blt01 36335368]$ /u01/app/oracle/em135/oms/OMSPatcher/omspatcher apply -analyze -property_file /u01/app/oracle/property_file/prop_file
-- We have provided SYS  username and password.
	
[oracle@blt01 36335368]$ /u01/app/oracle/em135/oms/OMSPatcher/omspatcher apply -property_file /u01/app/oracle/property_file/prop_file
...
Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:
...

The following artifact(s) have been successfully deployed:
...
OMSPatcher succeeded.

[oracle@blt01 36335368]$ cd ..

--Apply oneoff
[oracle@blt01 Setup]$ unzip p36634909_13500240430Bundle_Generic.zip
[oracle@blt01 36634909]$  /u01/app/oracle/em135/oms/OMSPatcher/omspatcher apply -property_file /u01/app/oracle/property_file/prop_file
...
Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:
...
OMSPatcher succeeded.

Now “One small step for man, one giant leap for dbakind”.

[oracle@blt01 Setup]$ cd 36635982/
[oracle@blt01 36635982]$ unzip p36635982_135000_Generic.zip

[oracle@blt01 36635982]$  /u01/app/oracle/em135/oms/OMSPatcher/omspatcher apply /u01/app/Setup/36635982 -spb_patch -property_file /u01/app/oracle/property_file/prop_file

JDK update log location :: /u01/app/oracle/em135/oms/cfgtoollogs/omspatcher/SPB_PATCH/jdk_update_Mon_Jun_10_11_21_10_2024.log

SPB JDK version :: 1.8.0.411,  Env JDK version :: 1.8.0.401

The Env version is lower as compared to JDK included in patch :: JDK will be updated
Updating the JDK now...
Stopping the OMS...
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
Oracle Management Server is Down
JVMD Engine is Down

New Java version is...
java version "1.8.0_411"
Java(TM) SE Runtime Environment (build 1.8.0_411-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.411-b25, mixed mode)

Updated Java Successfully

OPatch update log :: /u01/app/oracle/em135/oms/cfgtoollogs/omspatcher/SPB_PATCH/opatch_update_Mon_Jun_10_11_21_28_2024.log

The Env Opatch version :: 13.9.4.2.14

The SPB Opatch version :: 13.9.4.2.15

The Environment Opatch version is low as per OPatch included in Patch :: OPatch will be updated
Updating the Opatch now.......
Updated Opatch Successfully.......
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.9.5.20.0
OUI version        : 13.9.4.0.0
Running from       : /u01/app/oracle/em135/oms
Log file location  : /u01/app/oracle/em135/oms/cfgtoollogs/omspatcher/opatch2024-06-10_11-22-47AM_1.log

OMSPatcher log file: /u01/app/oracle/em135/oms/cfgtoollogs/omspatcher/SPB_PATCH/omspatcher_2024-06-10_11-22-47AM_apply.log


Performing prerequisites.....

The following sub-patches are already applied to the OMS system:

35965629 35430934 34153238 31657681 35024228 34065178 33093748


The following sub-patches will be applied to the OMS system:

36348444 34809489 1221421 36468190 32720458 36475175 36316422 36402397 34831198 36187017 36349529 36440005 36636070


Loaded SPB properties successfully

Stopping the oms....

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Running the command :: /u01/app/oracle/em135/oms/OPatch/opatch napply -oh /u01/app/oracle/em135/oms -phbasefile /u01/app/oracle/em135/oms/linux64_patchlist.txt -silent

Log file to be checked to get the list of applied patches : /u01/app/oracle/em135/oms/cfgtoollogs/opatch/opatch2024-06-10_11-23-09AM_1.log

Command ran successfully

Starting the oms....

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

Log file location: /u01/app/oracle/em135/oms/cfgtoollogs/omspatcher/SPB_PATCH/omspatcher_2024-06-10_11-22-47AM_apply.log

OMSPatcher succeeded.

At the patching stage you may track which patches are getting applied currently by tailing the file in $OMS_HOME/.patch_storage/NApply/2024-06-10_11-25-34AM/patchlist.txt

	User invokes OPatch via "NApply" to process the following patches:

	Patch ID: 1221421
	Patch Location: "/u01/app/Setup/36635982/binary_patches/coherence/generic/1221421"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/1221421_Mar_29_2024_17_54_16"

	Patch ID: 34809489
	Patch Location: "/u01/app/Setup/36635982/binary_patches/adf/generic/34809489"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/34809489_Nov_17_2022_23_24_13"

	Patch ID: 34831198
	Patch Location: "/u01/app/Setup/36635982/binary_patches/perl/linux64/34831198"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/34831198_May_11_2023_06_58_22"

	Patch ID: 36187017
	Patch Location: "/u01/app/Setup/36635982/binary_patches/rda/generic/36187017"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36187017_Mar_6_2024_07_05_44"

	Patch ID: 36316422
	Patch Location: "/u01/app/Setup/36635982/binary_patches/opss/generic/36316422"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36316422_Mar_29_2024_05_26_36"

	Patch ID: 36348444
	Patch Location: "/u01/app/Setup/36635982/binary_patches/adf/generic/36348444"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36348444_Feb_28_2024_22_57_35"

	Patch ID: 36349529
	Patch Location: "/u01/app/Setup/36635982/binary_patches/webcenter/generic/36349529"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36349529_Feb_28_2024_19_54_41"

	Patch ID: 36402397
	Patch Location: "/u01/app/Setup/36635982/binary_patches/owsm/generic/36402397"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36402397_Mar_14_2024_20_08_12"

	Patch ID: 36440005
	Patch Location: "/u01/app/Setup/36635982/binary_patches/wls/generic/36440005"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36440005_Mar_25_2024_11_18_02"

	Patch ID: 36468190
	Patch Location: "/u01/app/Setup/36635982/binary_patches/fmw_thirdparty/generic/36468190"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36468190_Apr_3_2024_11_34_38"

	Patch ID: 36475175
	Patch Location: "/u01/app/Setup/36635982/binary_patches/ohs/linux64/36475175"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36475175_Apr_3_2024_07_34_07"

	Patch ID: 36636070
	Patch Location: "/u01/app/Setup/36635982/binary_patches/wls/generic/36636070"
	Patch storage area: "/u01/app/oracle/em135/oms/.patch_storage/36636070_May_20_2024_03_29_25"

Patching OMS Agents demands extra effort. We’ll handle their patching individually. However, this isn’t related to Holistic Patching.

Hope it helps.