How to find which process caused cluster node reboot

Determining Which Process Caused Reboot
Log File Locations for Processes Causing Reboots.
• oclskd
– /log//client/oclskd.log
• ocssd
– /var/log/messages
– /log//cssd/ocssd.log
• cssdagent
– /log//agent/ohasd/oracss
dagent_root
• cssdmonitor
– /log//agent/ohasd/oracss
dmonitor_root
• hangcheck-timer
– /var/log/messages

Determining Which Process Caused Reboot
First, determine the time of the node reboot by using the uptime command and subtracting the
up time from the current system time. The reboot time will be used when examining log files.
When the OCSSD daemon is responsible for rebooting a node, a message similar to “Oracle
CSSD failure. Rebooting for cluster integrity” is written into the system messages log at
/var/log/messages. The cssd daemon log file that is located at
/log//cssd/ocssd.log may also contain messages similar to
"Begin Dump" or "End Dump" just before the reboot.
If hangcheck-timer is being used, it will provide message logging to the system messages log
when a node restart is initiated by the module. To verify whether this process was responsible
for the node reboot, examine the /var/log/messages file and look for an error message
similar to: "Hangcheck: hangcheck is restarting the machine."
Other useful log files include the Clusterware alert log in /log/
and the lastgasp log in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp.
If no indication of which process caused the reboot can be determined from these files,
additional debugging and tracing may need to be enabled.
Note: The oclsomon and the oprocd background processes have been eliminated in Oracle
Database 11g Release 2.

Standby redologs

Quote from documentation:

The standby redo logs are populated with redo information as fast as the primary redo logs, rather than waiting for the redo log to be archived and shipped to the standby database. This means that the standby redo log has more current information than the log apply mechanism because it took a "shortcut" and was written to the standby, bypassing the traditional archiving and FTP to the standby database.

Database compatible parameter is set to "11.2.0.0.0" and diskgroup database compatibility attribute is set to "11.2.0.2.0".

Issue:

Database compatible parameter is set to “11.2.0.0.0” and diskgroup database compatibility attribute is set to “11.2.0.2.0” while creating new database.
Solution
$ cd $ORACLE_HOME/assistants/dbca/templates/
[oracle@db01 templates]$ ls -lha
total 286M
drwxr-xr-x 2 oracle oinstall 4.0K Oct  5 19:03 .
drwxr-xr-x 6 oracle oinstall 4.0K Jun 30 13:01 ..
-rw-r--r-- 1 oracle oinstall 5.0K Oct  5 19:03 Data_Warehouse.dbc
-rw-r--r-- 1 oracle oinstall 5.0K Oct  5 18:56 Data_Warehouse.dbc.bk
-rwxr-xr-x 1 oracle oinstall  21M Sep 22  2011 example01.dfb
-rwxr-xr-x 1 oracle oinstall 1.5M Sep 22  2011 example.dmp
-rw-r--r-- 1 oracle oinstall 4.9K Sep 17  2011 General_Purpose.dbc
-rw-r--r-- 1 oracle oinstall  12K Feb 15  2010 New_Database.dbt
-rwxr-xr-x 1 oracle oinstall 9.3M Sep 22  2011 Seed_Database.ctl
-rwxr-xr-x 1 oracle oinstall 254M Sep 22  2011 Seed_Database.dfb
Edit template
Find
   <InitParamAttributes>
      <InitParams>
         <initParam name="db_name" value=""/>
         <initParam name="dispatchers" value="(PROTOCOL=TCP) (SERVICE={SID}XDB)"/>
         <initParam name="audit_file_dest" value="{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/adump"/>

Enable DDL Audit

ALTER SYSTEM SET enable_ddl_logging=TRUE
The result will go to
$ORACLE_BASE/diag/rdbms/db1/DB1/alert
log.xml

ORA-1113 when open database

Issue:
alter database open
ORA-1113 signalled during: alter database open...
Starting ORACLE instance (normal)
Solution:
startup mount
ALTER DATABASE RECOVER  database;
or
Check backup mode
        -- Hot backup status
        SELECT  COUNT(*) FROM v$backup WHERE status ='ACTIVE';
        -- Tablespaces in hot backup mode
        SELECT  distinct ddf.tablespace_name tablespace_name
        FROM    v$backup vb, DBA_DATA_FILES ddf
        WHERE   vb.status ='ACTIVE' and vb.file#  = ddf.file_id;
ALTER TABLESPACE (<tbs_name from the query before>) END BACKUP ;

Agent is Running but Not Ready

Issue
[oracle@db1 bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/EMbase/oracle/MW/agent11g/db1.local
Agent binaries    : /u01/app/oracle/product/EMbase/oracle/MW/agent11g
Agent Process ID  : 24910
Parent Process ID : 24865
---------------------------------------------------------------
Agent is Running but Not Ready
Workout
1. emctl stop agent
2. ps -ef | grep emagent
        kill all the process
3. emctl start agent
        Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
        Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
        Starting agent ........................................................................................................................... started but not ready.
Solution
emctl unsecure agent
emctl secure agent

How to convert to cluster database

Convert to cluster DB, make database clustered
1. copy init<>.ora and orapwd<> to another host
2. save spfile to init<>.ora
3. create UNDOTBS2,..
4. add logfiles for thread 2,..
5. assign UNDOTBS for each instance
6. assign instance_number for each instance;