Troubleshooting Exadata Storage Server Disk Failures

CacheContentLost State and Disk Failures on Exadata Servers

After a great weekend, we came to office and performed our daily health checks like every Monday. One of our storage server(cell) of Exadata X2-2 X4270 M2 had lost 11 ASM disks of total 34 ASM disks. We struck it lucky, all databases were still up despite all the lost.

Lets examine what happened to our cell server. When i checked the mailbox, i saw an alert mail from the problematic cell stating that “Disk controller was hung. Cell was power cycled”. It looks like cell disk controller was not performing well (may be a bug or a peak moment) and forced server to reboot. But normally reboots does not end up with disk losses.

I started with checking with cells physicaldisk status.

What I got from output was; we had one flashdisk and one harddisk failure (disk number 3) and also one harddisk was in import failure status(disk number 7). But that did not explain 11 ASM disks failure. It should had been 6 ASM disks according to output. There should be something more.

I continued with checking grid disks.

5 grid disks related with two physicaldisks (disk number 0 and disk number 11) were in “cacheContentLost” status.  I checked Oracle Support for the griddisks with “cacheContentLost” status

Doc Id 2346075.1 was related with our problem. Document was clear and explaining “the steps to recover the grid disks with cacheContentLost state.

When write-back flashcache is active on storage cells, in a flashdisk failure, the grid disks cached by the failed ones can be storing stale data. If the flash disk failure occurs while the Exadata storage software is running, a resilvering operation is started to resynchronize the stale blocks from the other storage servers.

But if the flashdisk failure happens while the storage software is not running or while rebooting phase of the cell, the resilvering operation is not started and the grid disks will be labeled to ‘cacheContentLost’ state. The grid disks stay offline to prevent the databases from accessing the stale data.

In our state, disk controller was hung and it ended up with rebooting the server. On reboot phase, flash disk failure occurred and grid disks stuck in “cacheContentLost”. Our team checked gv$asm_operation view for on going rebalance operations, there were no rows. Asmdisks related with that griddisk were already dropped, disk repair time had already passed. 

We decided to recreate the grid disks which are in “cacheContentLost” state to make them visible in ASM. 

After that commands, now we have 1 harddisk with import failure, 1 harddisk with failure and 1 flashdisk with 4 fmods in a failed state. We opened SRs for the flashdisk and the harddisk with failure.  We changed them with the spare new ones we had. Normally, No additional steps are required to re-create the cell disks or grid disks for flashdisk and the harddisk replacement.

Now let’s continue with our case, we had only one harddisk left with problems. We are only missing three griddisks and asmdisks. That disk was in import failure status. We executed the commands below to check harddisk information.

Foreign state is not looking good,  i will try to change foreing state of that harddisk. The commands below are executed for clearing foreign state and reconfiguring raid on that harddisk.

Griddisks for that disk were still in “not present” state. We had decided to go for the reenable command for that physicaldisk. The commands for reenable are as follows.

Now everything is perfect again. It was really a manic Monday. After that case, not to experience a same situation again, we also decided to update our Exadata servers image to the latest one. The issue has not happened again yet.

Hope it helps.