Upgrade Database 11.1.0.7 to 12.1.0.2 for EBS R12.1.3

DB version : 11.1.0.7.0

EBS: 12.1.3

Now I am going to upgrade my DB to 12.1.0.2


1) Stop all Application services and put the application in maintenance mode.

2) Below application patches need to applied.


Patch 8796558
Patch 12923944
Patch 13473483
Patch 16289505 - 12c Release 1 interoperability patch for Release 12.1
Patch 18843706 - apply only if you have applied a Recommended Patch Collection (RPC) patch
Patch 19558309

I will use adpatch utility to apply all the mentioned patches.


3) Install 12.1.0.2 binary in silent mode

Create a directory for 12c db oracle home
[oracle@ebs122 tech_st 07:01 PM]$pwd
/ebs121/oracle/DEV121/db/tech_st
[oracle@ebs122 tech_st 07:01 PM]$ls -ltr
total 4
drwxr-xr-x. 77 oracle oinstall 4096 Dec  1 18:00 11.1.0
[oracle@ebs122 tech_st 07:01 PM]$mkdir 12.1.0
[oracle@ebs122 tech_st 07:01 PM]$ls -ltr
total 8
drwxr-xr-x. 77 oracle oinstall 4096 Dec  1 18:00 11.1.0
drwxr-xr-x.  2 oracle oinstall 4096 Dec  3 19:01 12.1.0

Download the 12c database binaries and unzip and install them.

[oracle@ebs122 IO_Patches_R12.1.3_12cDB 07:06 PM]$ls -ltr linuxamd64*
-rwxrwx---. 1 root vboxsf 1014530602 Mar 14  2018 linuxamd64_12102_database_2of2.zip
-rwxrwx---. 1 root vboxsf 1673544724 Mar 14  2018 linuxamd64_12102_database_1of2.zip
The location where the database zip was unzipped.Please edit the response file and make changes according to the environment. We are doing silent installation using the response file.

cd /media/sf_Downloads/IO_Patches_R12.1.3_12cDB/database/response
cp db_install.rsp db_install_new.rsp
edit  db_install_new.rsp

Make sure the below values are set based on your environment.
--------------------------------------------
-- force to install only database software
--------------------------------------------
oracle.install.option=INSTALL_DB_SWONLY

--------------------------------------------
-- set your hostname
--------------------------------------------
ORACLE_HOSTNAME=ebs122.lab

--------------------------------------------
-- set unix group for oracle inventory
--------------------------------------------
UNIX_GROUP_NAME=oinstall

--------------------------------------------
-- set directory for oracle inventory
--------------------------------------------
INVENTORY_LOCATION=/ebs121/oraInventory

--------------------------------------------
-- set oracle home for binaries
--------------------------------------------
ORACLE_HOME=/ebs121/oracle/DEV121/db/tech_st/12.1.0

--------------------------------------------
-- set oracle home for binaries
--------------------------------------------
ORACLE_BASE=/ebs121/oracle

--------------------------------------------
-- set version of binaries to install
-- EE - enterprise edition
--------------------------------------------
oracle.install.db.InstallEdition=EE

DECLINE_SECURITY_UPDATES=true
--------------------------------------------
-- specify extra groups for database management
--------------------------------------------
oracle.install.db.DBA_GROUP=dba
oracle.install.db.OPER_GROUP=dba
oracle.install.db.BACKUPDBA_GROUP=dba
oracle.install.db.DGDBA_GROUP=dba
oracle.install.db.KMDBA_GROUP=dba


once the response file is created. Start the installation of Oracle 12c binaries

cd /media/sf_Downloads/IO_Patches_R12.1.3_12cDB/database
./runInstaller -silent -responseFile /media/sf_Downloads/IO_Patches_R12.1.3_12cDB/database/response/db_install_new.rsp -showProgress

[oracle@ebs122 database 07:32 PM]$./runInstaller -silent -responseFile /media/sf_Downloads/IO_Patches_R12.1.3_12cDB/database/response/db_install_new.rsp -showProgress 
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 5398 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 15992 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-12-03_07-32-41PM. Please wait ...[oracle@ebs122 database 07:32 PM]$You can find the log of this install session at:
 /ebs121/oraInventory/logs/installActions2020-12-03_07-32-41PM.log

Prepare in progress.
..................................................   8% Done.

Prepare successful.

Copy files in progress.
..................................................   13% Done.
..................................................   18% Done.
..................................................   23% Done.
..................................................   28% Done.
..................................................   33% Done.
..................................................   39% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.
..............................
Setup files successful.
..........
Setup Inventory in progress.

Setup Inventory successful.
..................................................   80% Done.
..........
Finish Setup successful.
The installation of Oracle Database 12c was successful.
Please check '/ebs121/oraInventory/logs/silentInstall2020-12-03_07-32-41PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
	1. /ebs121/oracle/DEV121/db/tech_st/12.1.0/root.sh



..................................................   100% Done.
Successfully Setup Software.

[oracle@ebs122 database 07:41 PM]$exit
logout
[root@ebs122 ~ 07:41 PM]$/ebs121/oracle/DEV121/db/tech_st/12.1.0/root.sh
Check /ebs121/oracle/DEV121/db/tech_st/12.1.0/install/root_ebs122.lab_2020-12-03_19-41-52.log for the output of root script
[root@ebs122 ~ 07:41 PM]$cat /ebs121/oracle/DEV121/db/tech_st/12.1.0/install/root_ebs122.lab_2020-12-03_19-41-52.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /ebs121/oracle/DEV121/db/tech_st/12.1.0
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
4) Create a temporary 12c environment file

12c.env

export ORACLE_HOME=/ebs121/oracle/DEV121/db/tech_st/12.1.0

export PATH=$ORACLE_HOME/bin:$PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib

export CLASS_PATH=$ORACLE_HOME/jlib

export ORACLE_SID=DEV121

export PERL5LIB=$ORACLE_HOME/perl/lib/5.14.1:$ORACLE_HOME/perl/lib/site_perl/5.14.1

5)Source 12c DB env

and run

perl $ORACLE_HOME/nls/data/old/cr9idata.pl

Known error:

perl: symbol lookup error: /ebs121/oracle/DEV121/db/tech_st/12.1.0/perl/lib/5.14.1/x86_64-linux-thread-multi/auto/Tie/Hash/NamedCapture/NamedCapture.so: undefined symbol: Perl_xs_apiversion_bootcheck

Solution:

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/nls/data/old/cr9idata.pl


Add below in 12c.env file 

 ORA_NLS10=$ORACLE_HOME/nls/data/9idata

[oracle@ebs122 ~ 07:55 PM]$. 12c.env
[oracle@ebs122 ~ 07:55 PM]$perl $ORACLE_HOME/nls/data/old/cr9idata.pl
Creating directory /ebs121/oracle/DEV121/db/tech_st/12.1.0/nls/data/9idata ...
Copying files to /ebs121/oracle/DEV121/db/tech_st/12.1.0/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /ebs121/oracle/DEV121/db/tech_st/12.1.0/nls/data/9idata!
6) Apply below DB patches on 12c Oracle Home using opatch.

Patch 6880880- the latest OPatch version for 12.1.0.
Patch 19382851
Patch 19393542
Patch 19591608
Patch 19627012
Patch 19649152
Patch 20204035
Patch 22660003
Patch 22731026
Patch 23645516

export PATH=$ORACLE_HOME/OPatch:$PATH

Verify all patches are applied using 

opatch lsinventory

7) Make sure application and database listener services are down.

8) Source 11g home and connect to sqlplus

Drop SYS.ENABLED$INDEXES (conditional)

SQL> drop table sys.enabled$indexes;

Remove the MGDSYS schema (conditional)

$ sqlplus "/ as sysdba" @?/md/admin/catnomgdidcode.sql

SQL> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

9)Copy the Pre-Upgrade Information Tool script preupgrd.sql and utluppkg.sql from the Oracle Database 12c Release 1 (12.1) to home

cd <12c_ORACLE_HOME>/rdbms/admin
cp preupgrd.sql utluppkg.sql $HOME


execute utluppkg.sql and preupgrd.sql
SQL> @utluppkg.sql

Package created.


Package body created.

SQL> @preupgrd.sql






Loading Pre-Upgrade Package...



***************************************************************************
Executing Pre-Upgrade Checks in DEV121...
***************************************************************************


      ************************************************************

		  ====>> ERRORS FOUND for DEV121 <<====

 The following are *** ERROR LEVEL CONDITIONS *** that must be addressed
		    prior to attempting your upgrade.
	    Failure to do so will result in a failed upgrade.


 1) Check Tag:	  INVALID_SYS_TABLEDATA
    Check Summary: Check for invalid (not converted) table data
    Fixup Summary:
     "UPGRADE Oracle supplied table data prior to the database upgrade."
    +++ Source Database Manual Action Required +++


 2) Check Tag:	  PURGE_RECYCLEBIN
    Check Summary: Check that recycle bin is empty prior to upgrade
    Fixup Summary:
     "The recycle bin will be purged."

	   You MUST resolve the above errors prior to upgrade

      ************************************************************

      ************************************************************

	      ====>> PRE-UPGRADE RESULTS for DEV121 <<====

ACTIONS REQUIRED:

1. Review results of the pre-upgrade checks:
 /ebs121/oracle/DEV121/db/tech_st/11.1.0/cfgtoollogs/DEV121/preupgrade/preupgrade.log

2. Execute in the SOURCE environment BEFORE upgrade:
 /ebs121/oracle/DEV121/db/tech_st/11.1.0/cfgtoollogs/DEV121/preupgrade/preupgrade_fixups.sql

3. Execute in the NEW environment AFTER upgrade:
 /ebs121/oracle/DEV121/db/tech_st/11.1.0/cfgtoollogs/DEV121/preupgrade/postupgrade_fixups.sql

      ************************************************************

***************************************************************************
Pre-Upgrade Checks in DEV121 Completed.
***************************************************************************

***************************************************************************
***************************************************************************

10)  Run-Preupgrade fixup

SQL> @/ebs121/oracle/DEV121/db/tech_st/11.1.0/cfgtoollogs/DEV121/preupgrade/preupgrade_fixups.sql
Pre-Upgrade Fixup Script Generated on 2020-12-03 21:16:56  Version: 12.1.0.2 Build: 006
Beginning Pre-Upgrade Fixups...
Executing in container DEV121

**********************************************************************
Check Tag:     DEFAULT_PROCESS_COUNT
Check Summary: Verify min process count is not too low
Fix Summary:   Review and increase if needed, your PROCESSES value.
**********************************************************************
Fixup Returned Information:
WARNING: --> Process Count may be too low

     Database has a maximum process count of 200 which is lower than the
     default value of 300 for this release.
     You should update your processes value prior to the upgrade
     to a value of at least 300.
     For example:
        ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE
     or update your init.ora file.
**********************************************************************


**********************************************************************
Check Tag:     OCM_USER_PRESENT
Check Summary: Check for OCM schema
Fix Summary:   Drop the ORACLE_OCM user.
**********************************************************************
Fixup Succeeded
**********************************************************************


**********************************************************************
Check Tag:     INVALID_SYS_TABLEDATA
Check Summary: Check for invalid (not converted) table data
Fix Summary:   UPGRADE Oracle supplied table data prior to the database upgrade.
**********************************************************************
Fixup Succeeded
**********************************************************************


**********************************************************************
Check Tag:     REMOVE_DMSYS
Check Summary: Check for existance of DMSYS schema
Fix Summary:   The DMSYS schema is removed as part of the upgrade.
**********************************************************************
Fixup Returned Information:
WARNING: --> "DMSYS" schema exists in the database

     The DMSYS schema (Oracle Data Mining) will be removed
     from the database during the database upgrade.
     All data in DMSYS will be preserved under the SYS schema.
     Refer to the Oracle Data Mining User's Guide for details.
**********************************************************************


**********************************************************************
Check Tag:     AMD_EXISTS
Check Summary: Check to see if AMD is present in the database
Fix Summary:   Manually execute ORACLE_HOME/oraolap/admin/catnoamd.sql script to remove OLAP.
**********************************************************************
Fixup Returned Information:
INFORMATION: --> OLAP Catalog(AMD) exists in database

     Starting with Oracle Database 12c, OLAP Catalog component is desupported.
     If you are not using the OLAP Catalog component and want
     to remove it, then execute the 
     ORACLE_HOME/olap/admin/catnoamd.sql script before or 
     after the upgrade.
**********************************************************************


**********************************************************************
Check Tag:     PURGE_RECYCLEBIN
Check Summary: Check that recycle bin is empty prior to upgrade
Fix Summary:   The recycle bin will be purged.
**********************************************************************
Fixup Succeeded
**********************************************************************


**********************************************************************
Check Tag:     JOB_QUEUE_PROCESS
Check Summary: Check JOB_QUEUE_PROCESSES value
Fix Summary:   Review and increase or remove the setting of job_queue_processes
**********************************************************************
Fixup Returned Information:
WARNING: --> job_queue_processes set too low

     This parameter must be removed or updated to a value greater
     than 4 (default value if not defined is 1000) prior to upgrade.
     Not doing so will affect the running of utlrp.sql after the upgrade

     Update your init.ora or spfile to make this change.
**********************************************************************


**********************************************************************
                      [Pre-Upgrade Recommendations]
**********************************************************************

                        *****************************************
                        ********* Dictionary Statistics *********
                        *****************************************

Please gather dictionary statistics 24 hours prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:
    EXECUTE dbms_stats.gather_dictionary_stats;

^^^ MANUAL ACTION SUGGESTED ^^^


                        *****************************************
                        *********** Hidden Parameters ***********
                        *****************************************

Please review and remove any unnecessary hidden/underscore parameters prior
to upgrading.  It is strongly recommended that these be removed before upgrade
unless your application vendors and/or Oracle Support state differently.
Changes will need to be made in the init.ora or spfile.

           ********    Existing Hidden Parameters   ********

_b_tree_bitmap_plans = FALSE
_disable_fast_validate = TRUE
_fast_full_scan_enabled = FALSE
_index_join_enabled = FALSE
_like_with_bind_as_equality = TRUE
_optimizer_autostats_job = FALSE
_sort_elimination_cost_ratio = 5
_sqlexec_progression_cost = 2147483647
_system_trig_enabled = TRUE
_trace_files_public = TRUE

^^^ MANUAL ACTION SUGGESTED ^^^


           **************************************************
                ************* Fixup Summary ************

 3 fixup routines were successful.
 4 fixup routines returned INFORMATIONAL text that should be reviewed.

**************** Pre-Upgrade Fixup Script Complete *********************

PL/SQL procedure successfully completed.

Some manual steps to be performed.

SYSAUX tablespace has to extended as per preupgrade.log file
set below parameter in init file and restart the database.
PROCESSES=300 

job_queue_processes=4
 EXECUTE dbms_stats.gather_dictionary_stats;

Comment these values:

 _b_tree_bitmap_plans = FALSE
_disable_fast_validate = TRUE
_fast_full_scan_enabled = FALSE
_index_join_enabled = FALSE
_like_with_bind_as_equality = TRUE
_optimizer_autostats_job = FALSE
_sort_elimination_cost_ratio = 5
_sqlexec_progression_cost = 2147483647
_system_trig_enabled = TRUE
_trace_files_public = TRUE

SQL> EXECUTE dbms_stats.gather_dictionary_stats;

11) Copy admin directory from 11G_HOME/network to 12c_HOME/network

[oracle@ebs122 ~ 09:40 PM]$. DEV121_ebs122.env
[oracle@ebs122 ~ 09:40 PM]$cd /ebs121/oracle/DEV121/db/tech_st/11.1.0/network/admin
[oracle@ebs122 admin 09:41 PM]$ls -ltr
total 12
-rw-r--r--. 1 oracle oinstall  187 May  8  2007 shrept.lst
drwxr-xr-x. 2 oracle oinstall 4096 Nov 26  2008 samples
drwxr-xr-x. 2 oracle oinstall 4096 Dec  1 18:06 DEV121_ebs122
[oracle@ebs122 admin 09:41 PM]$cp -r DEV121_ebs122 /ebs121/oracle/DEV121/db/tech_st/12.1.0/network/admin/
Update Oracle_Home path from 12c_home/network/admin files. Start listener from 12c home

Make sure 11g DB entry is there in /etc/oratab

Source 12c.env file

unset ORA_TZFILE

and launch ./dbua

Select the values as per screenshots below














If any XML parsing error is seen during the upgrade, kindly ignore it..





The database has been successfully upgraded

12) Verify invalid objects

SQL> select count(*) from dba_objects where status='INVALID'     
  2  ;

  COUNT(*)
----------
	48


13) Copy $APPL_TOP/admin/adgrants.sql  from the application server node to the database server node. 
Run as below

[applmgr@ebs122 ~ 01:55 AM]$cp $APPL_TOP/admin/adgrants.sql /tmp
ON DB server
$ sqlplus "/ as sysdba" @/tmp/adgrants.sql apps

14) Grant create procedure privilege on CTXSYS

Copy $AD_TOP/patch/115/sql/adctxprv.sql from the administration server node to the database server node. Use SQL*Plus to connect to the database as APPS and run the script using the following command:

$ sqlplus apps/[APPS password] @adctxprv.sql \
    [SYSTEM password] CTXSYS


15)Grant create procedure privilege on CTXSYS

Copy $AD_TOP/patch/115/sql/adctxprv.sql from the administration server node to the database server node. Use SQL*Plus to connect to the database as APPS and run the script using the following command:

$ sqlplus apps/[APPS password] @adctxprv.sql \
    [SYSTEM password] CTXSYS



    sqlplus apps/apps @adctxprv.sql manager CTXSYS

16) Set CTXSYS parameter
Use SQL*Plus to connect to the database as SYSDBA and run the following command:

$ sqlplus "/ as sysdba"
SQL> exec ctxsys.ctx_adm.set_parameter('file_access_role', 'public');


17) Validate Workflow ruleset
On the administration server node, use SQL*Plus to connect to the database as APPS and run the $FND_TOP/patch/115/sql/wfaqupfix.sql script using the following command:

$ sqlplus apps/apps @wfaqupfix.sql APPLSYS APPS

18) Gather statistics for SYS schema

Copy $APPL_TOP/admin/adstats.sql from the administration server node to the database server node. Note that adstats.sql has to be run in restricted mode. Use SQL*Plus to connect to the database as SYSDBA and use the following commands to run adstats.sql in restricted mode:

$ sqlplus "/ as sysdba"
SQL> alter system enable restricted session;
SQL> @adstats.sql 
$ sqlplus "/ as sysdba"
SQL> alter system disable restricted session;
SQL> exit;

19) Run FND_CONC cleanup
sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean;

20) Recreate appsutil and copy to DB server Oracle Home

[applmgr@ebs122 ~ ]$cd $APPL_TOP
[applmgr@ebs122 ~ ]$. APPSDEV121_ebs122.env 
[applmgr@ebs122 ~ ]$perl $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at /ebs121/apps/DEV121/inst/apps/DEV121_ebs122/admin/log/MakeAppsUtil_12011758.log
output located at /ebs121/apps/DEV121/inst/apps/DEV121_ebs122/admin/out/appsutil.zip
MakeAppsUtil completed successfully.
[applmgr@ebs122 ~ 05:58 PM]$


Copy the apputil.zip file to Database oracle home On DB-Tier:

Source the DB environment and start the listener.

[oracle@ebs122 ~ ]$. 12c.env
[oracle@ebs122 ~ ]$cd $ORACLE_HOME
[oracle@ebs122 12.1.0 ]$mv appsutil.zip appsutil.zip_orig
[oracle@ebs122 12.1.0 ]$cp /ebs121/apps/DEV121/inst/apps/DEV121_ebs122/admin/out/appsutil.zip .

unzip -o appsutil.zip

Recreate Context File on DB -Tier

export PATH=$ORACLE_HOME/perl:$ORACLE_HOME/perl/lib:$ORACLE_HOME/perl/bin:$PATH
export TNS_ADMIN=$ORACLE_HOME/network/admin
cd $ORACLE_HOME/appsutil/bin
[oracle@ebs122 bin 02:46 AM]$perl adbldxml.pl appsuser=apps appspasswd=apps jtop=$ORACLE_HOME/jdk

Starting context file generation for db tier..
Using JVM from /ebs121/oracle/DEV121/db/tech_st/12.1.0/jdk/bin/java to execute java programs..

The log file for this adbldxml session is located at:
/ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/log/adbldxml_12080746.log

Could not Connect to the Database with the above parameters, Please answer the Questions below


Enter Hostname of Database server: ebs122

Enter Port of Database server: 1546

Enter SID of Database server: DEV121

Enter the value for Display Variable: ebs122:0.0

The context file has been created at:
/ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/DEV121_ebs122.xml


21) Run adconfig  on DB Tier

[oracle@ebs122 bin 07:55 AM]$perl adconfig.pl contextfile=/ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/DEV121_ebs122.xml
Enter the APPS user password: 

The log file for this session is located at: /ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/log/DEV121_ebs122/12080755/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
	Using ORACLE_HOME location : /ebs121/oracle/DEV121/db/tech_st/12.1.0
	Classpath                   : :/ebs121/oracle/DEV121/db/tech_st/12.1.0/jdbc/lib/ojdbc6.jar:/ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/java/xmlparserv2.jar:/ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/java:/ebs121/oracle/DEV121/db/tech_st/12.1.0/jlib/netcfg.jar

	Using Context file          : /ebs121/oracle/DEV121/db/tech_st/12.1.0/appsutil/DEV121_ebs122.xml

Context Value Management will now update the Context file

	Updating Context file...COMPLETED

	Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db121
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

22) Run autoconfig on application Tier.

rm -fr $INST_TOP/ora/10.1.3/j2ee/oacore/persistence/*
rm -fr $INST_TOP/ora/10.1.3/j2ee/oafm/persistence/*
rm -fr $INST_TOP/ora/10.1.3/j2ee/forms/persistence/*


vi $ORA_CONFIG_HOME/Apache/Apache/conf/httpd.conf


1. Locate the KeepAlive directive in httpd.conf

   KeepAlive On
   Timeout 300
2. Replace the KeepAlive directive in httpd.conf with

   KeepAlive Off
   Timeout 600

23) Apply patch 18039691 on the application layer.

24) Disable maintenance mode in Application and start services. Verify the services and the DB version.




Reference Notes:

Interoperability Notes EBS 12.0 or 12.1 with RDBMS 12cR1 (Doc ID 1524398.1)

If you like please follow and comment