Is it possible to get credentials from crs?
In last two posts, I have been talking about the catastrophic situation we have experienced which was triggered with the mirrored disk failures. Up until now, we restored OCR config and recreated lost ASM disk group which was hosting OCR before. but we could not be able to start the CRS on more than one node.
We got the error “CRS-5019: All OCR locations on ASM disk groups [DATA], and none of these disk groups are mounted”. We recreated asm password file. (We should have restore it from offline disk group.)
| [root@exadb02 trace]# vi /u01/app/oracle/diag/crs/exadb02/crs/trace/alert.log | |
| … | |
| 2020-08-06 11:24:10.239 [ORAROOTAGENT(278746)]CRS-5019: All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/oracle/diag/crs/exadb02/crs/trace/ohasd_orarootagent_root.trc". | |
| … | |
| [root@exadb02 trace]# tail -f /u01/app/oracle/diag/crs/exadb02/crs/trace/ohasd_orarootagent_root.trc | |
| … | |
| 2020-08-06 11:29:37.160 : USRTHRD:697435904: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3169 | |
| 2020-08-06 11:29:37.160 : USRTHRD:697435904: [ INFO] {0:5:3} [ora.storage] ORA-01017: invalid username/password; logon denied | |
| … | |
| ASMCMD> pwcreate –asm +DATA/orapwASM welcome_1 | |
| ASMCMD-9465: WARNING: passing password on command line is deprecated | |
| OPW-00010: Could not create the password file. This resource has a Password File. | |
| ASMCMD-9454: could not create new password file | |
| ASMCMD> pwdelete –asm | |
| OPW-00022: The password file does not exist. | |
| ASMCMD-9462: could not delete password file | |
| ASMCMD> pwcreate –asm +DATA/orapwASM welcome_1 | |
| ASMCMD-9465: WARNING: passing password on command line is deprecated |
It was not enough to recreate it. There were some missing internal users.
According to the “Doc ID 2341753.1, The users used in Flex ASM“, CRSUSER__ASM_001 user is needed by crsd and it should have sysasm privilege, so we gave crs what it needed.
| We have this privileged database users in a healthy cluster : | |
| ASMCMD> lspwusr | |
| Username sysdba sysoper sysasm | |
| SYS TRUE TRUE TRUE | |
| CRSUSER__ASM_001 TRUE FALSE TRUE | |
| ASMSNMP TRUE FALSE FALSE | |
| What do we have : | |
| ASMCMD> lspwusr | |
| Username sysdba sysoper sysasm | |
| SYS TRUE TRUE FALSE | |
| We used below commands to fix it. | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –grant sysasm SYS | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –add CRSUSER__ASM_001 | |
| Enter password: ********* —> I used welcome1 | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –grant sysasm CRSUSER__ASM_001 | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –grant sysdba CRSUSER__ASM_001 | |
| [oracle@exadb01 ~]$ asmcmd lspwusr | |
| Username sysdba sysoper sysasm | |
| SYS TRUE TRUE TRUE | |
| CRSUSER__ASM_001 TRUE FALSE TRUE | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –add ASMSNMP | |
| Enter password: ********* | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –grant sysdba ASMSNMP | |
| [oracle@exadb01 ~]$ asmcmd lspwusr | |
| Username sysdba sysoper sysasm | |
| SYS TRUE TRUE TRUE | |
| CRSUSER__ASM_001 TRUE FALSE TRUE | |
| ASMSNMP TRUE FALSE FALSE |
We defined CRSUSER__ASM_001 user’s password on our own, but that was not the proper way. It is an internal user which is created at the grid installation part and its password is given internally.
We used ocrdump utility to view OCR and OLR contents by writing the content to a file to detect CRSUSER__ASM_001 user credential path and retrieved CRSUSER__ASM_001 user password according to the “Doc ID 2139591.1, ODA: CRS Could Not Start on Node Due to Invalid ASM Credentials for The “crsuser__asm_001″ Clusterware User” and modified it.
| [oracle@exadb01 ~]$ ocrdump /tmp/ocr.dmp | |
| PROT-310: Not all keys were dumped due to permissions. | |
| [oracle@exadb01 ~]$ vi /tmp/ocr.dmp | |
| … | |
| [SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001] | |
| ORATEXT : 8956aa1aaa66aa46aaaa2222a895aa5a:oracle | |
| … | |
| [oracle@exadb01 ~]$ crsctl get credmaint -path /ASM/Self/8956aa1aaa66aa46aaaa2222a895aa5a -credtype userpass -id 0 -attr passwd -local | |
| ZbV9Zx7F8BKxM2ZD4gUnKMjaH5Hnk | |
| [oracle@exadb01 ~]$ asmcmd orapwusr –modify CRSUSER__ASM_001 | |
| Enter password: ***************************** |
After modifying the CRSUSER__ASM_001 user password, CRS started normally.
We raised IRON MAN again.
We also changed backup location of asm password file location and that’s the commands we used.
| ASMCMD> pwcopy +DATA/orapwasm +RECO/orapwasm_backup | |
| copying +DATA/orapwasm -> +RECO/orapwasm_backup | |
| [oracle@exadb01 dbs]$ srvctl config asm | |
| ASM home: <CRS home> | |
| Password file: +DATA/orapwasm | |
| Backup of Password file: +DATA/orapwASM_backup | |
| ASM listener: LISTENER | |
| ASM instance count: ALL | |
| Cluster ASM listener: ASMNET1LSNR_ASM | |
| [oracle@exadb01 dbs]$ srvctl modify asm -pwfilebackup +RECO/orapwasm_backup | |
| [oracle@exadb01 dbs]$ srvctl config asm | |
| ASM home: <CRS home> | |
| Password file: +DATA/orapwasm | |
| Backup of Password file: +RECO/orapwasm_backup | |
| ASM listener: LISTENER | |
| ASM instance count: ALL | |
| Cluster ASM listener: ASMNET1LSNR_ASM |
That was all the story to heal cluster and asm disk groups, later we configured dataguard databases again.
Hope it helps.
Osman’s DBlog 
Leave your comment