For other A-Team articles about BICS, click here

Introduction

Oracle Data Integrator (ODI) is a comprehensive data integration platform that covers most data integration scenarios.  It has long been possible to use ODI to load data into BI Cloud Service (BICS) environments, that use Database as a Service (DBaaS) as the underlying database.

The recent 12.2.1.2.6 release of ODI added the ability to load data into BICS environments based on a Schema Service Database.  ODI does this by using the BICS REST API.

This article will walk through the following steps to set up ODI to load data into the BICS schema service database through this method:

• Downloading latest version of ODI
• Configuring the physical and logical connection to BICS in ODI
• Loading BICS knowledge modules
• Reverse engineering BICS model
• Create a simple mapping
• Importing the BICS certificate into the trust store for the standalone agent

This article will not cover the installation and setup of ODI.  The assumption is that a 12.2.1.2.6 environment has been stood up and is working correctly.  For details on how to install and configure ODI, see this document.

 

Main Article

Download The Latest Version of Oracle Data Integrator

Download and install the latest version of ODI from OTN through this link.

 

Configure and Test Connection to BICS

This article will walk through one (of the several) methods to set up the BICS connection with a Physical and Logical connection.  For more details on topology and other approaches, see this document.

1. In ODI studio, select the ‘Topology‘ tab, and expand out ‘Technologies‘ under the Physical Architecture section

2. Scroll down to the ‘Oracle BI Cloud Service‘ entry, right click and select ‘New Data Server‘

3. Give the Data Server a name, and enter the BICS Service URL, as well as the user credentials and Identity Domain.

The syntax for the URL is:

https://service-identity_domain.analytics.data_center.oraclecloud.com

This URL can be obtained from the BICS instance, by taking the first part of the URL up to ‘oraclecloud.com’

Note – the Data Loader path will default to /dataload/v1, leave this.

4. Save the Data Server.  ODI will give you an informational warning about needing to register at least one physical schema.  Click ‘OK‘.

5. Test the connection by selecting ‘Test Connection‘

For the time being, use the ‘Local (No Agent)‘ option.

NOTE – Once configuration has been completed, the ODI Agent where the execution will be run should also be tested.  It is likely that additional configuration will need to be carried out – this is covered in the last section of this article ‘Importing the BICS certificate into the trust store for the standalone agent’.

If the credentials and URL have been entered correctly, a notification similar to the following should be displayed.  If an error is displayed, trouble-shoot and resolve before continuing.

TIP :  

ODI studio’s local agent uses the JDK’s certificate store, whereas the Standalone Agent does not.  It is therefore possible – and quite likely – that while the local agent will provide a successful Test Connection, the Standalone agent will produce and error similar to the following:

oracle.odi.runtime.agent.invocation.InvocationException: oracle.odi.core.exception.OdiRuntimeException: javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

To resolve this, the BICS Certificate needs to be added to the trust store used by the Standalone agent.  These steps are covered later in this article in the section ‘Importing Certificate into Trust Store‘

 

6. Right click on the Data Server created in the step 2, and select ‘New Physical Schema‘

ODI has the ability to load to both the Database Objects (Tables) in the Schema Service Database, and also Data Sets.

This loading option is chosen in the ‘Target Type‘ dropdown.  The selection then associates the appropriate REST Operations for ODI to connect.  Note – once the target type has been chosen and saved, it cannot be changed.

7. In this example the Target Type of Table is selected.

8. Save the Physical Schema.

Because we haven’t associated this with a Logical Architecture yet, the following warning will be shown.  Click OK to complete the save.

9. Expand out the Logical Architecture section of Topology, and then right click on ‘Oracle BI Cloud Service‘ and create a ‘New Logical Schema‘

10. In the configuration window, give the Logical Schema a descriptive name, and associate your context(s) with the physical schema that was created in steps 6-8.  Save the changes.

11. Repeat the steps from 6 on if you need to create an additional connection to load Data Sets

 

Load BICS Knowledge Modules

ODI uses 2 different Knowledge Modules for BICS:

– a reverse knowledge module (RKM) called RKM Oracle BI Cloud Service, and

– an integration knowledge module (IKM) called IKM SQL to Oracle BI Cloud Service.

 

1. In ‘Designer‘ expand your project and the knowledge modules and see if the KMs are already available.

If they are – continue to the ‘Reverse Engineering‘ section of this article.

2. If the KMs are not shown, right click on the Knowledge Modules section and select ‘Import Knowledge Modules‘

Browse to a path similar to this to find the import directory.

/u01/oracle/ODI12c/odi/sdk/xml-reference

3. In the import wizard, select the 2 BICS KMs, and then select ‘OK’ to load them.

TIP :  

If you already have used ODI for other integration tasks, you may be tempted to use existing Knowledge Modules.  Please note that the IKM SQL to Oracle BI Cloud Service does not support loading the Oracle SDO_GEOMETRY data type column to the BICS target table.

Oracle BI Cloud Service cannot be used as the staging area, and does not support incremental update or flow/static check. Therefore, the following KMs will not work with the Oracle BI Cloud Service technology:

RKM SQL (JYTHON)

LKM File to SQL

CKM SQL

IKM SQL Incremental Update

IKM SQL Control Append

LKM SQL to SQL (JYTHON)

More details can be found in this document.

 

Reverse Engineer BICS

Reverse-engineering is the process that populates the model in ODI, by retrieving metadata from the data server containing the data structures.

 

1. Create a new model in Designer, by selecting the ‘New Model‘ option as shown below

2. In the Definition tab, given the model a name, select the ‘Oracle BI Cloud Service‘ as the technology, and select the Logical Schema created previously.

3. In the Reverse Engineer tab, leave the logical agent set to ‘Local (No Agent)‘, and select the RKM Oracle BI Cloud Service knowledge module.  Then save the changes.

TIP :  

At the time of writing this article, there is a bug in the reverse knowledge module that will present an error if tables in the BICS environment contain non-standard characters.

An error like the following may be generated:

ODI-1590: The execution of the script failed.
Caused By: org.apache.bsf.BSFException: exception from Groovy: oracle.odi.runtime.rest.SnpsRSInvocationException: ODI-30163: REST tool invocation failed with response code : 500. URL : https://businessintelltrialXXXX-usoracletrialXXXXX.analytics.us2.oraclecloud.com/dataload/v1/tables/APEX$TEAM_DEV_FILES

There is at least one Apex related table within BICS environments that has a non-standard character.  That table, as shown in the error above, is ‘APEX$TEAM_DEV_FILES’.

Until this issue is fixed, a workaround is required.

The simplest is to go into the Apex environment attached to the BICS environment, rename the APEX$TEAM_DEV_FILES table temporarily, run the Reverse Engineer process, and then rename the table back.

Another method is to use the ‘Mask’ import option box.  If you have a specific table(s) you need to reverse engineer, then enter the name followed by %

For instance, if there were 5 tables all starting ‘FACT….’, then a mask of ‘FACT%’ could be used to reverse engineer those 5 tables.

 

4. Select the ‘Reverse Engineer‘ action, and then ‘OK‘ to run the action.

5. This will start a session that can be viewed in the Operator.

6. Once the session has completed, expand the model to confirm that the database objects have been imported correctly.  As shown below, the tables in the BICS Schema Service database are now available as targets.

7. Expand the BICS individual database objects that you will load, and confirm within the attributes that the Datatypes have been set correctly.  Adjust where necessary and save.

 

Create Mapping

1. Within the ‘Mapping‘ sub-menu of the project, select ‘New Mapping‘

2. Drag in the source table from the source that will be loaded into BICS, and then the BICS target table, and link the two together.  For more information on how to create mappings, see this document.

TIP :  

The BICS API only allows data to be loaded, not ‘read’ or ‘selected’.  Because of this, BICS using the Schema Service Database CAN ONLY BE USED as a TARGET for ODI mappings.  It can not be used as a SOURCE.

 

3. Make sure the Target is using the IKM SQL to Oracle BI Cloud Service:

and that an appropriate loading KM is used:

4. Run the mapping, selecting the Local Agent

5. Confirm in the Operator that the mapping was successful.  Trouble-shoot an errors you find and re-run.

 

Importing Certificate into Trust Store

To operate, it is likely that the Standalone Agent will require the BICS certificate be added to its trust store.

These instructions will use Microsoft Explorer, although other browsers offer similar functionality.

1. In a browser, open the BICS /analytics portal, then click on the padlock icon.  This will open an information box, within which select ‘View certificates‘

2. In the ‘Details‘ tab, select the ‘Copy to File‘ option which will open an export wizard.

3. Select the ‘DER encoded binary‘ format and then ‘Next‘

4. Chose a path and file name for the certificate, then ‘Next‘, and on the final screen ‘Finish‘ to export the certificate.

 

TIP :  

This article will go through the steps needed to add this certificate to the DemoTrust.jks key store.  This should *ONLY* be followed for demonstration or test environments.  For production environments, follow best practice guidelines as outlined in this document.

 

5. Copy the certificate file created in the previous steps to a file system accessible by the host running the standalone ODI agent.

6. Set the JAVA_HOME to the path of the JDK used while installing the standalone agent, for example

export JAVA_HOME=/u01/oracle/jdk1.8.0_111/bin

7. Browse to the bin directory of the ODI Domain Home, in this test environment that path is as follows:

/u01/oracle/ODI12c/user_projects/domains/base_domain/bin

8. Run the ‘setODIDomainEnv‘ script.  In a linux environment this would be:

./setODIDomainEnv.sh

The DemoTrust.jks keystore used by the agent should be located in the following path:

$ORACLE_HOME/wlserver/server/lib

 

TIP :  

It is possible that there are a number of DemoTrust.jks key stores on the file system, so make sure the correct one is updated.  If this process fails to resolve the error with the Standalone Agent, search the file system and see if it is using a different trust store.

 

9. Browse to that directory and confirm the DemoTrust.jks file exists.  In that same directory – run the keytool command to import the certificate created earlier.

The syntax for the command is as follows, $CERTIFICATE referencing the name/path for the certificate file downloaded from the BICS environment through the browser, $ALIAS being a name for that, and $KEYSTORE the name/path of the key store.

keytool -importcert -file $CERTIFICATE -alias $ALIAS -keystore $KEYSTORE

In this example, the command would be:

keytool -importcert -file /u01/oracle/Downloads/BICS.cer -alias BICS -keystore DemoTrust.jks

the default password is DemoTrustKeyStorePassPhrase

10. Details of the certificate are displayed and a prompt to ‘Trust this certificate?’ is displayed.  Type ‘yes‘ and then hit enter.

If the import is successful, a confirmation that the certificate was added to the keystore is given.

11. Return to ODI and run the mapping, this time selecting the Standalone agent, and confirm it runs successfully.

Summary

This article walked through the steps to configure ODI to load data into the BICS schema service database through the BICS API

For other A-Team articles about BICS, click here.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission