03 July 2014

ERROR: failed to establish dependency between database and diskgroup resource

Issue:
ERROR: failed to establish dependency between database and diskgroup resource

has found in alert log

Explanation:
This means your database is not configured in clusterware. If you run command "crsctl stat res -t" you will not probably find it in the list.

Solution:
Register your database in clusterware like:
srvctl add database -d -o -p -a "," -r -s
Example:
srvctl add database -d standby -o /u01/app/oracle/product/11.2.0/db_1 -p /u01/app/oracle/product/11.2.0/db_1/dbs/spfilestandby.ora -a "DATA,FRA" -r PHYSICAL_STANDBY -s mount

13 May 2014

Oracle Heterogeneous service setup on Linux

on linux
1. install ODBS drivers
yum install unixODBC
yum install mysql-connector-odbc

2. check if driver is in /etc/odbcinst.ini
# Driver from the mysql-connector-odbc package
# Setup from the unixODBC-libs package
[MySQL]
Description     = ODBC for MySQL
Driver          = /usr/lib/libmyodbc3_r.so
Setup           = /usr/lib/libodbcmyS.so
Driver64        = /usr/lib64/libmyodbc3_r.so
Setup64         = /usr/lib64/libodbcmyS.so
FileUsage       = 1

3. add entry into /etc/odbc.ini
[MySQL]
Description     = MySQL test database
Trace       = Off
TraceFile   = stderr
Driver      = MySQL
SERVER      = 192.168.x.x
USER        = root
PASSWORD    = *******
PORT        = 3306
DATABASE    = test

4. test ODBC connection
$ isql -v MySQL

5. configure Oracle heterogeneous service
$ cp /u01/app/oracle/product/11.2.0/db_1/hs/admin/initdg4odbc.ora /u01/app/oracle/product/11.2.0/db_1/hs/admin/initMySQL.ora

6. edit /u01/app/oracle/product/11.2.0/db_1/hs/admin/initMySQL.ora
$ cat ../../hs/admin/initMySQL.ora | grep -v ^"#"

HS_FDS_CONNECT_INFO = MySQL
HS_FDS_TRACE_LEVEL = OFF
HS_FDS_SHAREABLE_NAME = /usr/lib/libodbc.so
HS_LANGUAGE=american_america.we8mswin1252
set ODBCINI=/etc/odbc.ini

7. configure listener
/u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
SID_LIST_LISTENER=
...
      (SID_DESC=
         (SID_NAME=MySQL)
         (ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1)
         (PROGRAM=dg4odbc)
      )
...
  )
 ...
restart listener

8. add entry into /u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora
...
mysql  =
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521))
    (CONNECT_DATA=(SID=MySQL))
    (HS=OK)
  )
...

9. connect into DB and create database link

SQL> create public database link MY_MYSQL connect to "root" identified by "*******" using 'mysql';

10. test select
SQL> select * from table1@my_mysql;

23 April 2014

Oracle Result Cache

Oracle Result Cache

Available only in Enterprise Edition

Documentation quote:
The result cache stores the results of SQL queries and PL/SQL functions in an area called Result Cache Memory in the shared pool. When these queries and functions are executed repeatedly, the results are retrieved directly from the cache memory. This results in a faster response time. The cached results stored become invalid when data in the dependent database objects is modified. The use of the result cache is database-wide decision. Result cache itself is instance specific and can be sized differently on different instances. To disable the result cache in a cluster, you must explicitly set the RESULT_CACHE_MAX_SIZE initialization parameter to 0 during every instance startup.

How to enable:
check the settings:
SQL> show parameter result_cache
NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
client_result_cache_lag              big integer 3000
client_result_cache_size             big integer 0
result_cache_max_result              integer     10
result_cache_max_size                big integer 2G
result_cache_mode                    string      AUTO
result_cache_remote_expiration       integer     0

AUTO is meaningless (you can find this setting in SE as default)

RESULT_CACHE_MODE
Property Description
Parameter type String
Syntax RESULT_CACHE_MODE = { MANUAL | FORCE }
Default value MANUAL
Modifiable ALTER SESSION, ALTER SYSTEM
Basic No

SQL> alter system set result_cache_mode=FORCE scope=memory;

SQL> alter system set result_cache_max_size=2G scope=memory;
--> Note: Result Cache is a part of SGA.

How to monitor:
SQL> select DBMS_RESULT_CACHE.status from dual;
STATUS
----------------------------
ENABLED

Generate report:
SQL> set serveroutput on
SQL> execute DBMS_RESULT_CACHE.MEMORY_REPORT(TRUE);
R e s u l t   C a c h e   M e m o r y   R e p o r t
[Parameters]
Block Size          = 0 bytes
Maximum Cache Size  = 0 bytes (0 blocks)
Maximum Result Size = 0 bytes (0 blocks)
[Memory]
Total Memory = 5184 bytes [0.005% of the Shared Pool]
... Fixed Memory = 5184 bytes [0.005% of the Shared Pool]
....... Memory Mgr = 132 bytes
....... Cache Mgr  = 152 bytes
....... Bloom Fltr = 2K bytes
....... State Objs = 2852 bytes
... Dynamic Memory = 0 bytes [0.000% of the Shared Pool]

PL/SQL procedure successfully completed.




28 March 2014

RMAN-04014: startup failed: ORA-00845: MEMORY_TARGET not supported on this system

Issue description
when I did standby creation on other host by using rman duplicate command I've got an error like:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 03/27/2014 15:41:59
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-04014: startup failed: ORA-00845: MEMORY_TARGET not supported on this system

was really surprised because the standby was configured with less SGA memory than actual primary like

32Gb - SGA* on Production
4 Gb - SGA* on Standby
* - as for MEMORY_TARGET parameter

Reason for the Issue
the problem was in system configuration

[root@sby01 ~]# df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/vg_sby01-lv_root   53G   18G   34G  35% /
tmpfs                              3.0G  30M  2.9G   1% /dev/shm
/dev/vda1                          485M  100M  360M  22% /boot
10.1.1.5:/u02                     734G  494G  240G  68% /u02

Workaround
[root@sby01 ~]# vi /etc/fstab

#
# /etc/fstab
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_sby01-lv_root /                       ext4    defaults        1 1
UUID=ae17aee-972b-4f91-806b-f3e33d29a0 /boot                   ext4    defaults        1 2
/dev/mapper/vg_mtksby01p2-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults,size=8g        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

I've allocated 8Gb for tmpfs

[root@sby01 ~]# mount -o remount /dev/shm

31 October 2013

Oracle Operating System Statistics


-- Operating System Statistics
1. CPU Statistics
V$OSSTAT
                - view captures machine-level information in the database, making it easier for you to determine if hardware-level resource issues exist.
V$SYSMETRIC_HISTORY
                - view shows a one-hour history of the Host CPU Utilization metric, a representation of percentage of CPU usage at each one-minute interval
V$SYS_TIME_MODEL
                - view supplies statistics on the CPU usage by the Oracle database. Using both sets of statistics enable you to determine whether the Oracle database or other system activity is the cause of the CPU problems
--
2. Memory
                This command below includes a great deal of detail, including the size of the mapped segment, how much if it is RSS (Real memory), how much of that is shared, the sizes and permissions of the segment. It is the output above that can be ‘compared’ over time to see if a given process is growing, or leaking memory.
                $ pmap -x $$
                where $$ - process ID

3. Disk I/O Statistics
                - Consumer group
                                V$IOSTAT_CONSUMER_GROUP view captures I/O statistics for all consumer groups that are part of the currently enabled resource plan
                - Database file
                                I/O statistics of database files that are or have been accessed are captured in the V$IOSTAT_FILE view
                                select * from V$IOSTAT_FILE
                - Database function
                                I/O statistics for database functions (such as the LGWR and DBWR) are captured in the V$IOSTAT_FUNCTION view.
                                select * from V$IOSTAT_FUNCTION
4. Network Statistics
                V$IOSTAT_NETWORK - network I/O statistics
                select * from V$IOSTAT_NETWORK

Operating System Data Gathering Tools
Table shows the various tools for gathering operating statistics on *NIX.
                Component        | UNIX Tool
                CPU                       | sar, vmstat, mpstat, iostat
                Memory                              | sar, vmstat
                Disk                        | sar, iostat
                Network                              |netstat

15 October 2013

Oracle streams propagation issue ORA-22636

Issue
SQL> SELECT DESTINATION_DBLINK,
  2         STATUS,
  3         ERROR_DATE,
  4         ERROR_MESSAGE
  5    FROM DBA_PROPAGATION;

DESTINATION_DBLINK
----------------------------------------------------------------
ERROR_MESSAGE
----------------------------------------------------------------
TO_FORCL.COM
ORA-22636: Message 22636 not found;  product=RDBMS; facility=ORA

Diagnostic
Run on both sites
SQL> select * from v$timezone_file;

FILENAME                VERSION
-------------------- ----------
timezlrg_14.dat              14
--
SQL> select * from v$timezone_file;

FILENAME                VERSION
-------------------- ----------
timezlrg_11.dat              11

the versions must be the same

Workaround
Update DST file. Follow the document:
Updating the RDBMS DST version in 11g Release 2 (11.2.0.1 and up) using DBMS_DST (Doc ID 977512.1)

lsnodes.bin: struct size 0

Issue
cluster nodes list
when trying to get a node list with lsnodes
lsnodes.bin: struct size 0
Solution
Try
cd $ORACLE_GRID (/u01/app/11.2.0/grid/bin)
./olsnodes


29 July 2013

sqlplus: error while loading shared libraries


Issue
sqlplus: error while loading shared libraries: /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: cannot restore segment prot after reloc: Permission denied

Solution
1.
                [root@linux_pc ~]# cat /etc/selinux/config
                # This file controls the state of SELinux on the system.
                # SELINUX= can take one of these three values:
                #       enforcing - SELinux security policy is enforced.
                #       permissive - SELinux prints warnings instead of enforcing.
                #       disabled - SELinux is fully disabled.
                SELINUX=enforcing
                # SELINUXTYPE= type of policy in use. Possible values are:
                #       targeted - Only targeted network daemons are protected.
                #       strict - Full SELinux protection.
                SELINUXTYPE=targeted

                # SETLOCALDEFS= Check local definition changes
                SETLOCALDEFS=0
                [root@linux_pc ~]# cp /etc/selinux/config /etc/selinux/config.bak
                [root@linux_pc ~]# vi /etc/selinux/config
                [root@linux_pc ~]# cat /etc/selinux/config
                # This file controls the state of SELinux on the system.
                # SELINUX= can take one of these three values:
                #       enforcing - SELinux security policy is enforced.
                #       permissive - SELinux prints warnings instead of enforcing.
                #       disabled - SELinux is fully disabled.
                SELINUX=disabled
                # SELINUXTYPE= type of policy in use. Possible values are:
                #       targeted - Only targeted network daemons are protected.
                #       strict - Full SELinux protection.
                SELINUXTYPE=targeted

2.
                echo 0 >/selinux/enforce

25 July 2013

PRVF-0002 : Could not retrieve local nodename

Issue:

While Oracle 12c Installation on Linux (my case OEL6 update 4) you could get an error.

Solution 1:

1. check hostname:
cat /etc/sysconfig/network | grep HOSTNAME
HOSTNAME = test.localdomain

2. check hosts:
cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


3. change /etc/hosts like:
127.0.0.1 test test.localdomain
::1 test test.localdomain


4. Run installer

Solution 2:

1. setup system variable like

$ ORACLE_HOSTNAME=test.localdomain
$ export ORACLE_HOSTNAME

07 June 2013

How to find which process caused cluster node reboot


Determining Which Process Caused Reboot
Log File Locations for Processes Causing Reboots.
• oclskd
/log//client/oclskd.log
• ocssd
– /var/log/messages
/log//cssd/ocssd.log
• cssdagent
/log//agent/ohasd/oracss
dagent_root
• cssdmonitor
/log//agent/ohasd/oracss
dmonitor_root
• hangcheck-timer
– /var/log/messages

Determining Which Process Caused Reboot
First, determine the time of the node reboot by using the uptime command and subtracting the
up time from the current system time. The reboot time will be used when examining log files.
When the OCSSD daemon is responsible for rebooting a node, a message similar to “Oracle
CSSD failure. Rebooting for cluster integrity” is written into the system messages log at
/var/log/messages. The cssd daemon log file that is located at
/log//cssd/ocssd.log may also contain messages similar to
"Begin Dump" or "End Dump" just before the reboot.
If hangcheck-timer is being used, it will provide message logging to the system messages log
when a node restart is initiated by the module. To verify whether this process was responsible
for the node reboot, examine the /var/log/messages file and look for an error message
similar to: "Hangcheck: hangcheck is restarting the machine."
Other useful log files include the Clusterware alert log in /log/
and the lastgasp log in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp.
If no indication of which process caused the reboot can be determined from these files,
additional debugging and tracing may need to be enabled.
Note: The oclsomon and the oprocd background processes have been eliminated in Oracle
Database 11g Release 2.

Standby redologs



Quote from documentation:

The standby redo logs are populated with redo information as fast as the primary redo logs, rather than waiting for the redo log to be archived and shipped to the standby database. This means that the standby redo log has more current information than the log apply mechanism because it took a "shortcut" and was written to the standby, bypassing the traditional archiving and FTP to the standby database.

24 November 2012

Database compatible parameter is set to "11.2.0.0.0" and diskgroup database compatibility attribute is set to "11.2.0.2.0".


Issue:

Database compatible parameter is set to “11.2.0.0.0” and diskgroup database compatibility attribute is set to “11.2.0.2.0” while creating new database.
Solution
$ cd $ORACLE_HOME/assistants/dbca/templates/
[oracle@db01 templates]$ ls -lha
total 286M
drwxr-xr-x 2 oracle oinstall 4.0K Oct  5 19:03 .
drwxr-xr-x 6 oracle oinstall 4.0K Jun 30 13:01 ..
-rw-r--r-- 1 oracle oinstall 5.0K Oct  5 19:03 Data_Warehouse.dbc
-rw-r--r-- 1 oracle oinstall 5.0K Oct  5 18:56 Data_Warehouse.dbc.bk
-rwxr-xr-x 1 oracle oinstall  21M Sep 22  2011 example01.dfb
-rwxr-xr-x 1 oracle oinstall 1.5M Sep 22  2011 example.dmp
-rw-r--r-- 1 oracle oinstall 4.9K Sep 17  2011 General_Purpose.dbc
-rw-r--r-- 1 oracle oinstall  12K Feb 15  2010 New_Database.dbt
-rwxr-xr-x 1 oracle oinstall 9.3M Sep 22  2011 Seed_Database.ctl
-rwxr-xr-x 1 oracle oinstall 254M Sep 22  2011 Seed_Database.dfb
Edit template
Find
   <InitParamAttributes>
      <InitParams>
         <initParam name="db_name" value=""/>
         <initParam name="dispatchers" value="(PROTOCOL=TCP) (SERVICE={SID}XDB)"/>
         <initParam name="audit_file_dest" value="{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/adump"/>

Enable DDL Audit


ALTER SYSTEM SET enable_ddl_logging=TRUE
The result will go to
$ORACLE_BASE/diag/rdbms/db1/DB1/alert
log.xml

ORA-1113 when open database


Issue:
alter database open
ORA-1113 signalled during: alter database open...
Starting ORACLE instance (normal)
Solution:
startup mount
ALTER DATABASE RECOVER  database;
or
Check backup mode
        -- Hot backup status
        SELECT  COUNT(*) FROM v$backup WHERE status ='ACTIVE';
        -- Tablespaces in hot backup mode
        SELECT  distinct ddf.tablespace_name tablespace_name
        FROM    v$backup vb, DBA_DATA_FILES ddf
        WHERE   vb.status ='ACTIVE' and vb.file#  = ddf.file_id;
ALTER TABLESPACE (<tbs_name from the query before>) END BACKUP ;

22 November 2012

Agent is Running but Not Ready


Issue
[oracle@db1 bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/EMbase/oracle/MW/agent11g/db1.local
Agent binaries    : /u01/app/oracle/product/EMbase/oracle/MW/agent11g
Agent Process ID  : 24910
Parent Process ID : 24865
---------------------------------------------------------------
Agent is Running but Not Ready
Workout
1. emctl stop agent
2. ps -ef | grep emagent
        kill all the process
3. emctl start agent
        Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
        Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
        Starting agent ........................................................................................................................... started but not ready.
Solution
emctl unsecure agent
emctl secure agent

How to convert to cluster database



Convert to cluster DB, make database clustered
1. copy init<>.ora and orapwd<> to another host
2. save spfile to init<>.ora
3. create UNDOTBS2,..
4. add logfiles for thread 2,..
5. assign UNDOTBS for each instance
6. assign instance_number for each instance;

11 June 2012

ORA-600 [kkfi.qbcvfr]

Issue: ORA-600 [kkfi.qbcvfr]

Environment:
RDBMS: 11.2.0.2
OS: Linux 5.5

Error in alert log:
ORA-600 [kkfi.qbcvfr]

Solution:
Patch 9965278

Possible solution (temporary):
alter system/session set "_replace_virtual_columns" = false;
alter system/session set "_disable_function_based_index"=true;
alter system/session set "_fix_control"="6754080:off";
alter system/session set "_fix_control"="8706652:off";

Exadata Storage srvers CellCli and dcli commands


-- Exadata Starage Servers
CELLCLI
-- use root user to do the work
-- cell configuration files
cellinit.ora, cellip.ora
-- on database server
cellinit.ora - identifies the storage network interface on the database server
        cat /etc/oracle/cell/network-config/cellinit.ora
cellip.ora - identifies the Exadata cells that are accessible to the database server
        cat /etc/oracle/cell/network-config/cellip.ora
-- ASM view to check disk configuration
        V$ASM_DISKGROUP
      
-- support & diagnostic utility
/opt/oracle.SupportTools
--  usefull commands
cellcli> list cell
cellcli> alter cell restart services all -- restarting RS, CELLSRV, and MS services
cellcli> alter cell {SHUTDOWN | RESTART | STARTUP } services {RS | MS | CELLSRV | ALL}
cellcli> create celldisk all harddisk {interleaving='normal_redundancy'}
cellcli> list celldisk
         CD_00_xdb1cel03         normal
         CD_01_xdb1cel03         normal
cellcli> create griddisk all prefix=data, size=300G
cellcli> list griddisk
         DATA_XDB1_CD_00_xdb1cel03       active
         DATA_XDB1_CD_01_xdb1cel03       active
-- to crate disk group on flash         
cellcli> drop flashcache
cellcli> create flashcache all size=100G
cellcli> create griddisk all flashdisk prefix=flash
-- to use all flash cache
cellcli> drop flashcache
cellcli> create flashlog all
cellcli> create flashcache all
------------------------------------------------------------------------------------------------------------------
-- DCLI
-- /opt/oracle/cell/cellsrv/bin/dcli or /usr/local/bin/dcli
dcli -g cell_list date
dcli -c cell01,cell02,cell03 cellcli -e list cell
dcli -g cell_list -x cellclicommands.scl
dcli -g mydbservers -l root -x dbwork.sh
-- cell_list is a file with the list of cells
------------------------------------------------------------------------------------------------------------------
-- IORM  I/O Resource Manager
cellcli> list iormplan detail
         name:                   xdb1cel03_IORMPLAN
         catPlan:
         dbPlan:
         objective:              off
         status:                 active
cellcli> alter iormplan objective='balanced'
-- {balanced | off | low_latency | high_throughput | auto}
-- interdatabase Plane Example
cellcli> alter iormplan dbplan =((name=sales_prod, level=1, allocation=80),
                                                (name=finance_prod, level=1, allocation=20),
                                                (name=sales_dev, level=2, allocation=100),
                                                (name=sales_test, level=3, allocation=50),
                                                (name=other, level=3, allocation=50)
                                                ),
                                catplan=''
-- interdatabase Plane and Database Role
cellcli> alter iormplan dbplan =((name=sales1, level=1, allocation=30, role=primary),
                                                (name=sales2, level=1, allocation=35, role=primary),
                                                (name=sales3, level=2, allocation=20, role=standby),
                                                (name=sales4, level=2, allocation=25, role=standby),
                                                (name=other, level=3, allocation=50)
                                                ),
                                catplan=''
-- Flash Memory usage
cellcli> alter iormplan dbplan =((name=oltp, level=1, allocation=80, flashCache=on, flashLog=on),
                                                (name=dss, level=1, allocation=20 limit=50, flashCache=off, flashLog=on),
                                                (name=other, level=2, allocation=100, flashCache=off, flashLog=off),
-- catplan usage
        catplan=((name=INTERACTIVE, level=1, allocation=60),
                         (name=BATCH, level=1, allocation=40)
                        )
-- Activation iormplan
cellcli> alter iormplan active

12 January 2012

Alert log from DB side

--
-- view alert log from DB side
--
-- 0. get info
select * from dba_directories
select * from v$instance
-- 1. create dir
create directory ALERT_LOG as 'G:\SBLSYS\ORADBA\TRACE\BDUMP'
-- 2. create external table
create table alert_log_f (txt varchar2(2000))
organization external (type oracle_loader -- oracle_datapump <-for 10g -- oracle_loader <-for Oracle 9i
default directory ALERT_LOG
--      access parameters (
--          --records delimited by newline
--          nologfile nobadfile
--          fields terminated by 2000 ltrim
--      )
  location('alert_orcl.log'))
reject limit unlimited;
drop table alert_log_f
-- 3. check
select * from alert_log_f where lower(txt) like '%ora-%'

11 January 2012

How to enable Automatic Memory Management in Oracle 11G

How to enable Automatic Memory Management in Oracle 11G
1. Set MEMORY_TARGET as (memory_target = sga_target + max(pga_aggregate_target, maximum PGA allocated))
        ALTER SYSTEM SET MEMORY_TARGET = mM SCOPE = SPFILE;
2. Set parameter MEMORY_MAX_TARGET
        ALTER SYSTEM SET MEMORY_MAX_TARGET = nM SCOPE = SPFILE;
Note:   1. n > m
                2. if don't specify MEMORY_TARGET it sets to zero by default
                        and you can manipulate its value dynamicaly not exceeding MEMORY_MAX_TARGET
3. Set SGA_TARGET to zero
        ALTER SYSTEM SET SGA_TARGET = 0;
4. Set PGA_AGGREGATE_TARGET to zero
        ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 0;
       
Note:
The preceding steps instruct you to set SGA_TARGET and PGA_AGGREGATE_TARGET to zero
so that the sizes of the SGA and instance PGA are tuned up and down as required,
without restrictions. You can omit the statements that set these parameter values to zero
and leave either or both of the values as positive numbers.
In this case, the values act as minimum values for the sizes of the SGA or instance PGA.