Opatch rollback fails – .patch_storage directory is missing

4 Apr 2026 22:32

—

There are moments in a DBA’s life where something small quietly turns into something painfully big. This was one of them.

I was working on an Oracle 12.1.0.2 environment. One of those systems that has been around for quite some time, still running, still “working,” and therefore… still trusted.

The task was actually to update the database to the latest available release, patch 34057733 (Database Bundle Patch 12.1.0.2.220719), as we encountered bug 27873364 ORA-600[ktliReadStreamAdvanceBuf_2] when Select on UNIFIED_AUDIT_TRAIL .The task was actually updating it to the latest available release 34057733 (Database Bundle Patch 12.1.0.2..220719)

			
Following error occurred while executing a query from unified_audit_trail.
SQL> select * from unified_audit_trail;
    select * from unified_audit_trail
    *
    ERROR at line 1:
    ORA-12801: error signaled in parallel query server PPA7, instance XXX
    ORA-00600: internal error code, arguments: [ktliReadStreamAdvanceBuf_2]...

		

Before applying the patch, it was thoroughly tested in a test environment that we believed closely mirrored production. We defined the playbook, identified the patches that needed to be rolled back due to conflicts, and documented the steps for applying the patch and running datapatch.

A very common operation. Or at least, it should be.

Failure at the First Step – rollback failed with error code 73

The rollback process started, prompted for confirmation, and then failed during the make step. This was not after applying something new.

The failure occurred before any new patching activity had even begun.

At that point, the Oracle Home was already in trouble.

Below is the command I executed along with its output:

			
[oracle@dbnode1 setup] $ORACLE_HOME/OPatch/opatch rollback -id 28542455
Oracle Interim Patch Installer versiob 12.2.0.1.17
Copyright (c) 2026, Oracle Corporation. All rights reserved.
...
Patches will be rolled back in the following order:
  28542455
--------------------------------------------------------------------
...
Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with:Y
Rolling back patch 28542455...
RollbackSession rolling back interim patch '28542455' from OH '/u01/app/oracle/product/12.1.0.2/dbhome1'
Patching component oracle.rdbms, 12.1.0.2.0...
RollbackSession removing interim patch '28542455' from inventory
Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome1"....'/u01/app/oracle/product/12.1.0.2/dbhome1/lib//libserver12.a(ktvt.o): In function `ktvtGenerateQuery':
ktvt.c:(text.unlikekly+0x1ec0): undefined reference to `qcscop'
ktvt.c:(text.unlikekly+0x210f): undefined reference to `qcscop'
ktvt.c:(text.unlikekly+0x22d3): undefined reference to `qcscop'
make: *** [/u01/app/oracle/product/12.1.0.2/dbhome1/rdbms/lib/oracle] Error 1
'
----------------------------------------------------------------
Failed to run make commands. Please contact Oracle Support.
Opatch failed to restore OH '/u01/app/oracle/product/12.1.0.2/dbhome1'. Consult OPatch document to restore the home manually before proceeding.
NRollback was not able to restore the home. Please invoke the following scripts:
  - restore.[sh,bat]
  - make.txt (Unix only)
to restore the ORACLE_HOME. 
UtilSession failed: Re-link fails on target "ioracle".
Log file location:
/u01/app/oracle/product/12.1.0.2/dbhome1/cfgtoollogs/opatch/opatch2026-03-23_13-35-35PM_1.log
OPatch failed with error code 73

		

That was all the output, completely clean, no warnings. Even though I had not specified a silent flag, it reported running in silent mode. There were no warnings before the error struck, and the Oracle Home ended up broken.

I investigated the log, but it contained no helpful messages. I also attempted to relink the Oracle Home using the relink all command, but that too failed.

.patch_storage directory was missing

At that stage, neither MOS nor a Google search yielded any helpful information. I decided to investigate what happens behind the scenes. In a successful rollback, Opatch actually restores files to their previous versions by retrieving them from the .patch_storage directory under $ORACLE_HOME.

When I checked the .patch_storage location, the surprise was revealed, the directory was missing from $ORACLE_HOME. Because this directory was absent, the Opatch session could not recover the old versions of the files, leaving the Oracle Home in a corrupted state.

The .patch_storage directory is essential for both patching and rollback operations. It contains:

Backup files of replaced binaries
Patch metadata
Inventory information required for rollback

Without it, Oracle cannot properly restore the previous state of the binaries, which explains why the make process failed and the Oracle Home became inconsistent.

The big question now: how do we fix a corrupted Oracle Home when .patch_storage is missing? What solutions are available?

Option 1: Restore from backup (Best option)

If you have a backup of the Oracle Home (which contains .patch_storage directory) , restoring it is the safest and most reliable solution.
For example, a backup of database home taken with:

tar -zcf oracle_home_backup.tar.gz $ORACLE_HOME

In my case, this was not an option.

Option 2: Copy from an identical Oracle Home

If another Oracle Home exists with the exact same version and patch level, you can copy it and then restore environment-specific files such as:

Network files (tnsnames.ora, sqlnet.ora etc )
$ORACLE_HOME/dbs contents (spfile, password file, etc.)
Environment/configuration files (.env files)

In my case, this was not possible because this was the last remaining home with that specific old version.

Option 3: Prepare/reinstall a fresh Oracle Home and reapply patches

Reinstall the Oracle Home and reapply all required patches (or use ApplyRU for a streamlined process).

However, in my case, this was also not feasible because most of the one-off patches applied earlier were no longer downloadable (they had been superseded).

Option 4: Recover from VM snapshot / clone

If running on a virtual machine, you may recover the entire environment to a clone, which you can restore the .patch_storage directory directly from there.

In my case, the environment was running on Exadata, so this option was not applicable.

Option 5: Build a new Oracle Home with the intended patches (my solution), following an out-of-place patching methodology similar to applying Release Updates.

Prepare a new Oracle Home with the same major version, but patched to a more recent RU/BP level. You can either:

Install a fresh home and patch it, or
Copy from an already prepared home.

Important point:
Ensure the sqlpatch directory is correctly populated. If needed, copy the patch apply/rollback sqlfiles (based on DBA_REGISTRY_SQLPATCH) so that datapatch can apply the SQL components properly.

Finally;

Copy required configuration files from the corrupted home. {Network files (tnsnames.ora, sqlnet.ora etc ), dbs contents (spfile, password file, etc.) }

In my case, this was the only viable solution and it worked. Start the database using the new Oracle Home.
and run datapatch to rollback and apply sql patches

Conclusion :

This incident was a hard reminder: before any patching or rollback, always verify that .patch_storage exists and is intact and never start without taking a full backup of the Oracle Home.

Hope it helps.