Introduction
This article outlines a configuration option for Oracle Java Cloud Service (JCS) in the Oracle Cloud to support switchover between datacenters to allow for ongoing availability. The configuration allows easy switchover between datacenters in case of scheduled maintenance and other outages. In this example the primary datacenter is US2 and secondary is EM2.
This example has been intentionally kept simple to allow easy understanding of the concept. The WebLogic configuration part that is stored on the file system will be synced between the two datacenters using rsync. For simplicity the entire WebLogic DOMAIN_HOME will be synchronized this allows all configuration changes on the WebLogic environment to be replicated. Oracle Data Guard is used to sync application data as well as the configuration data that is stored in the database. Only static application data is allowed to be stored on the file system, all other data has to be placed in the database to allow switchover without data loss. The usage of DNS aliases on all nodes make this solution work with minimal setup effort and no manual reconfiguration requirements in case of a switchover.
For this configuration the deployment has to be symmetrical to allow simplified switch and failover. This means the same number of VMs, Cluster etc. have to be deployed on both sides.
The following VMs are involved in this example – please note the are anonymised IP addresses.
| Short Name | Purpose | Internal IP | External IP | Data Centre |
| Loadbalancer | HAProxy Server | 10.0.0.1 | 129.0.0.1 | US2 |
| WL Node 1 | WebLogic Server | 10.148.1.1 | 129.0.1.1 | US2 |
| WL Node 2 | WebLogic Server | 10.196.1.2 | 140.0.1.2 | EM2 |
| DB Node 1 | Database Server | 10.148.2.1 | 129.0.2.1 | US2 |
| DB Node 2 | Database Server | 10.196.2.2 | 140.0.2.2 | EM2 |
This is the simplified architecture diagram for this example:
Creating Storage Container
In this example two individual storage containers are used to allow separate backups of both environments. More details can be found in this document.
Deploying Oracle Cloud – Database as a Service (DBaaS)
After creating the Storage Container the next step is to deploy DBaaS for both environments as a backend for the JCS Services. A separate DBaaS Service with the same name, e.g. “jcs-db” has to be created in both datacenters. For this example the “Oracle Database Cloud Services” option as Oracle Database 12c Release 1 with Enterprise Edition or better is recommended. The DBaaS Virtual Image option requires additional steps that are not discussed in this article. Oracle Database Standard Edition is not supported. For more details on deploying DBaaS see here.
The service name has to be identical between the two instances – here “jcs-db”. It is mandatory to reference the storage container that has been created in the corresponding data center. Please note the format “instance-id_domain/container” where instance is the name of the Oracle Storage Cloud Service instance, id_domain is the name of the identity domain, and container is the name of the container – e.g. Storage-rdb/jcsContainer. Select the SSH public key carefully and make sure to take a secure backup of the private key. More details can be found here.
Continue with the same deployment in the second identity domain. Make sure to select the same Service Name for the database, as shown below.
Deploying Oracle Cloud – Java Cloud Service (JCS)
In the next step create a separate JCS Services in both datacenters. For more details on deploying JCS see here.
For this example choose Service Level “Oracle Java Cloud Server” and Billing Frequency based on your preference. The Oracle Java Cloud Service – Virtual Image option requires additional steps that are not discussed in this article. Oracle WebLogic Server 12c (12.2.1.0) with Enterprise Edition is used – however the procedure can be applied to Oracle WebLogic 11g and 12c.
The Service Name can be chosen freely, however it has to be the same across both deployment. The same goes for the database configuration. It is possible to select a different PDB, if required as long as it is the same as in the second instance.
The second instance has to be created identical to the one created first. Make sure to select the correct storage container based on what has been created earlier – all other settings should remain the same.
Load Balancer Configuration
This examples is using the open source proxy HAProxy as an exemplary load balancer based on the simple configuration. In an enterprise production deployment this should be replaced by an enterprise grade DNS based load balancer or similar mechanism.
To install HAProxy simply login as opc user and use the commands below on the Oracle Cloud VM where you want to run the HAProxy. For production use this should not be one of the JCS or DB VMs running the workload.
sudo yum –y install haproxy
To configure HAProxy simply append the following lines to the /etc/haproxy/haproxy.conf. This configuration will route all traffic that is hitting the port 5555 on the HAProxy VM to the JCS Application server based on their availability. Load Balanced by Round Robin. This is Layer 7 Load Balancing – further details can be found here.
frontend http bind 0.0.0.0:5555 default_backend jcs-backend backend jcs-backend balance roundrobin mode http option httpchk GET /sample-app/index.html server jcs-1 10.148.1.1:8001 check server jcs-2 10.196.1.2:8001 check
Then start the HAProxy Service:
service haproxy start
Network Configuration
Open the Oracle Compute Service Console navigate to Network and Create a new Security IP List for the Load Balancer:
Continue by creating a security application for the JCS app that is supposed to be accessed via the load balancer. In this example the JCS Sample App is used.
Finish by creating a security rule – note that the Destination Security List jcs/wls/ora_ms is created during JCS provisioning. This target list represents the WebLogic Managed Server.
Should the HAProxy be deployed on Oracle Compute, make sure to open up the configured port. To achieve this create a new Security Application:
Then create a new Security Rule to allow Traffic to the HAProxy from the Clients. For simplicity public-internet is chosen in this example. Generally this should be minimized to the actual client IPs that will be using the application.
Next validate that you can open up the sample-app via the loadbalancer.
To test that the load balancing is working properly login to the Administration Console of the WebLogic instances and stop the sample application first on the first instance – validate that the application is still available and then move on to the next instance.
Creating host alias
The DNS configuration is crucial to get the environment to switch properly between the datacenter. The JCS alias for the nodes need to be resolvable to the local internal IP to get WebLogic to bind properly to the correct interfaces. Since the deployments are in different Identity Domains make sure to include the alias
Ensure that the database traffic is only routed to the corresponding database in the same datacenter by including the DNS alias in the /etc/hosts of the WebLogic VM.
The /etc/hosts should look similar to this:
WL Node 1:
WL Node 2:
Enable Configuration Synchronizations
SSH equivalence
In order to achieve secure communication between the WebLogic VMs it is recommended to establish SSH equivalence between the nodes. In order to achieve this login to the first instance and execute the following commands and copy the public key as shown below.
sudo su - ssh-keygen cat ~/.ssh/id_rsa.pub
The copied key needs to be pasted in the ~/.ssh/authorized_keys of the opc user on the second WebLogic server (WL Node 2). For more details see here.
Make sure the connections via SSH from the root to the opc user are working between the two instances both ways.
sudo su - ssh opc@10.196.1.2 sudo su - ssh opc@10.148.1.1 exit
RSYNC Configuration
Before the synchronization can be enabled all servers and the NodeManager should be stopped on the secondary site. Login as OPC user switch into the oracle user and execute the following. In this example this is done on WL Node 2.
/u01/app/oracle/middleware/oracle_common/common/bin/wlst.sh
nmConnect('weblogic', 'Welcome1#', '127.0.0.1', '5556','jcs_domain','/u01/data/domains/jcs_domain')
nmKill('jcs_doma_server_1')
nmKill('jcs_doma_adminserver')
exit()
kill -9 $(cat /u01/data/domains/jcs_domain/nodemanager/nodemanager.process.id)
Next remove the current configuration from the node so that the configuration from the primary site can be copied without issues.
mv /u01/data/domains/jcs_domain /u01/data/domains/jcs_domain.bkp
Then configure a cron job to enable copying the data between the sites. The domain directory should be copied into a staging directory – here /u01/data/domains/jcs_stage. This is done to avoid problems with partially completed copies. More information about cron jobs and the crontab can be found here. Alternatively other scheduling tools can be used. To edit the crontab simply run as opc user:
crontab -e
The crontab should look like the following, to sync the data from WL Node 1 to WL Node 2 every 10 minutes.
*/10 * * * * sudo rsync -avz --exclude "*.log" --log-file=/u01/data/domains/rsync.log --rsync-path="sudo rsync" opc@129.0.1.1:/u01/data/domains/jcs_domain /u01/data/domains/jcs_stage
Database Configuration
The database configuration for this example is similar to the configuration has been described in this white paper. Simply follow the steps outlined in the section Appendix A: Case Study Details matching the Service Names to what has been configured in this example as shown in the table below.
| Node | Service Name | Unique Name |
| DB1 | JCS | JCS |
| DB2 | JCS | JCSEM2 |
Switchover Execution
The following diagram shows the basic steps that are carried out as part of a switchover.
On WL Node 1 the following steps have to be carried out as user oracle to stop the WebLogic Server.
/u01/app/oracle/middleware/oracle_common/common/bin/wlst.sh
nmConnect('weblogic', 'Welcome1#', '127.0.0.1', '5556','jcs_domain','/u01/data/domains/jcs_domain')
nmKill('jcs_doma_server_1')
nmKill('jcs_doma_adminserver')
exit()
kill -9 $(cat /u01/data/domains/jcs_domain/nodemanager/nodemanager.process.id)
Add a # symbol in front of the crontab rsync job to disable further syncing of data to the remote site.
On Database Node 1 the following steps have to be carried out:
. oraenv ORACLE_SID = [ORCL] ? ORCL dgmgrl / switchover to jcsem2 exit
On WebLogic Node 2 the following steps have to be carried out to start the environment. Make sure to review the rsync log prior to executing these steps.
cp /u01/data/domains/jcs_stage/jcs_domain /u01/data/domains
cd /u01/data/domains/jcs_domain/nodemanager
/u01/data/domains/jcs_domain/bin/startNodeManager.sh &
/u01/app/oracle/middleware/oracle_common/common/bin/wlst.sh
nmConnect('weblogic', 'Welcome1#', '127.0.0.1', '5556','jcs_domain','/u01/data/domains/jcs_domain')
nmStart('jcs_doma_adminserver')
nmStart('jcs_doma_server_1')
exit()
Due to the failover configuration of the Load Balancer no other change is required. To enable WL Node 2 to be the Primary node for the Synchronization simply disable the cron job on WL Node 2 and enable the following job on WL Node 1.
*/10 * * * * sudo rsync -avz --exclude "*.log" --log-file=/u01/data/domains/rsync.log --rsync-path="sudo rsync" opc@ 10.148.1.1:/u01/data/domains/jcs_domain /u01/data/domains/jcs_stage
In case of a switchback the steps have to be carried out in the inverted order.
Next Steps
As discussed in the introduction this example is intentionally kept simple. One great option is to implement the copy jobs in Enterprise Manager. This allows for monitoring of the process and corrective actions. This is described in the Enterprise Manager Administration Guide. The switchover process can also be executed from Enterprise Manager.
All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission