Introduction

This post details a method of extracting and loading data from Oracle Field Service Cloud (OFSC) into the Oracle Business Intelligence Cloud Service (BICS).

A compelling reason to use such a method is when data is required that is not in the standard daily extract. Such data might be planning (future) data or data recently provided in new releases of the application.

This post uses SOAP web services to extract XML-formatted data responses. It also uses the PL/SQL language to wrap the SOAP extract, XML parsing commands, and database table operations in a Stored Procedure. It produces a BICS staging table and a staging view which can then be transformed into star-schema object(s) for use in modeling. The transformation processes and modeling are not discussed in this post.

Finally, an example of a database job is provided that executes the Stored Procedure on a scheduled basis.

The PL/SQL components are for demonstration purposes only and are not intended for enterprise production use. Additional detailed information, including the complete text of the PL/SQL procedure described, is included in the References section at the end of this post.

Rationale for Using PL/SQL

PL/SQL is the only procedural tool that runs on the BICS / Database Schema Service platform. Other wrapping methods e.g. Java, ETL tools, etc. require a platform outside of BICS to run on.

PL/SQL may also be used in a DBCS that is connected to BICS.

PL/SQL can utilize native SQL commands to operate on the BICS tables. Other methods require the use of the BICS REST API.

Note: PL/SQL is very good at showcasing functionality. However, it tends to become prohibitively resource intensive when deploying in an enterprise production environment. For the best enterprise deployment, an ETL tool such as Oracle Data Integrator (ODI) should be used to meet these requirements and more:

* Security

* Logging and Error Handling

* Parallel Processing and Performance

* Scheduling

* Code Re-usability and Maintenance

Using Oracle Database Cloud Service

Determining Security Protocol Requirements

If the web service requires a security protocol, key exchange or cypher not supported by the default BICS Schema Database Service, another Oracle Database Cloud Service (DBCS) may be used.

An example security protocol is TLS version 1.2 which is used by the OFSC web service accessed in this post.

Note: For TLSv1.2, specify a database version of 11.2.0.4.10 or greater, or any version of 12c. If the database is not at the required version, PL/SQL may throw the following error: ORA-29259: end-of-input reached

To detect what protocol a web service uses, open the SOAP WSDL page in a browser, click the lock icon, and navigate to the relevant security section. A Chrome example from an OFSC WSDL page is below:

Preparing the DBCS

If a DBCS other than the default Schema Service is used, the following steps need to be performed.

Create a BICS user in the database. The use of Jobs and the DBMS_CRPTO package shown in the example below are discussed later in the post. Example SQL statements are below:

— USER SQL
CREATE USER “BICS_USER” IDENTIFIED BY password
DEFAULT TABLESPACE “USERS”
TEMPORARY TABLESPACE “TEMP”
ACCOUNT UNLOCK;
— QUOTAS
ALTER USER “BICS_USER” QUOTA UNLIMITED ON USERS;
— ROLES
ALTER USER “BICS_USER” DEFAULT ROLE “CONNECT”,”RESOURCE”;
— SYSTEM PRIVILEGES
GRANT CREATE VIEW TO “BICS_USER”;
GRANT CREATE ANY JOB TO “BICS_USER”;
–OBJECT PERMISSIONS
GRANT EXECUTE ON SYS.DBMS_CRYPTO TO BICS_USER;

Create an entry in a new or existing Oracle database wallet for the trusted public certificate used to secure connections to the web service via the Internet. A link to the Oracle Wallet Manager documentation is included in the References section. Note the location and password of the wallet as they is used to issue the SOAP request.

The need for a trusted certificate is detected when the following error occurs: ORA-29024: Certificate validation failure.

An example certificate path found using Chrome browser is shown below. Both of these trusted certificates need to be in the Oracle wallet.

Preparing the Database Schema

Two objects need to be created prior to compiling the PL/SQL stored procedure.

The first is a staging table comprising a set of identical columns. This post uses a staging table named QUOTA_STAGING_TABLE. The columns are named consecutively as C01 through Cnn. This post uses 50 staging columns. The SQL used to create this table may be viewed here.

The second is a staging view named QUOTA_STAGING_VIEW built over the staging table. The view column names are the attribute names used in the API WSDL. The SQL used to create this view may be viewed here. The purpose of the view is to relate an attribute name found in the SOAP response to a staging table column based on the view column’s COLUMN_ID in the database. For example, if a response attribute name of bucket_id is detected and the COLUMN_ID of the corresponding view column is 3, then the staging table column populated with the attribute value would be C03.

Ensuring the Web Services are Available

To ensure that the web services are available in the required environment, type a form of the following URL into a browser:

https://hostname/soap/capacity/?wsdl

Note: If you are unable to reach the website, the services may not be offered or the URL may have changed. Discuss this with the service administrator.

Using API Testing Tools

The SOAP Request Envelope should be developed in an API testing tool such as SoapUI or Postman. The XPATH expressions for parsing should be developed and tested in an XPATH expression testing tool such as FreeFormatter. Links to these tools are provided in the References section.

Note: API testing tools such as SoapUI, FreeFormatter, Postman, and so on are third-party tools for using SOAP and REST services. Oracle does not provide support for these tools or recommend a particular tool for its APIs. You can select the tool based on your requirements.

Preparing the SOAP Request

This post uses the get_quota_data method of the Oracle Field Service Cloud Capacity Management API. Additional information about the API is included as a link in the References section.

Use a browser to open the WSDL page for this API. An example URL for the page is: https://hostname/soap/capacity/?wsdl. This page provides important information regarding the request and response envelopes used by the API.

The request envelope is comprised of the following sections. Note: To complete the envelope creation, the sections are concatenated together to provide a single request envelope. An example of a complete request enveloped may be viewed here.

Opening

The Opening section is static text as shown below:

<soapenv:Envelope xmlns:soapenv=”http://schemas.xmlsoap.org/soap/envelope/” xmlns:urn=”urn:toa:capacity”>
<soapenv:Header/>
<soapenv:Body>
<urn:get_quota_data>

User

The User section is dynamic and comprises the following components:

Now

The now component is the current time in the UTC time zone. An example is: <now>2016-12-19T09:13:10+00:00</now>. It is populated by the following command:

SELECT TO_CHAR (SYSTIMESTAMP AT TIME ZONE ‘UTC’, ‘YYYY-MM-DD”T”HH24:MI:SS”+00:00″‘ ) INTO V_NOW FROM DUAL;

Login

The login component is the user name.

Company

The company component is the company for which data is being retrieved.

Authorization String

The auth_string component is the MD5 hash of the concatenation of the now component with the MD5 hash of the user password. In pseudo-code it would be md5 (now + md5 (password)). It is populated by the following command:

SELECT
LOWER (
DBMS_CRYPTO.HASH (
V_NOW||
LOWER( DBMS_CRYPTO.HASH (V_PASSWORD,2) )
,2
)
)
INTO V_AUTH_STRING FROM DUAL;

Note: ‘2’ is the code for MD5.

An example is:

<auth_string>b477d40346ab40f1a1a038843d88e661fa293bec5cc63359895ab4923051002a,/auth_string>

Required Parameters

There are two required parameters: date and resource_id. Each may have multiple entries. However the sample procedure in this post allows only one resource id. It also uses just one date to start with and then issues the request multiple times for the number of consecutive dates requested.

In this post, the starting date is the current date in Sydney, Australia. An example is below:

<date>2016-12-21</date> <resource_id>Test_Resource_ID</resource_id>

The starting date and subsequent dates are populated by this command:

CASE WHEN P_DATE IS NULL
THEN SELECT TO_CHAR (SYSTIMESTAMP AT TIME ZONE ‘Australia/Sydney’, ‘YYYY-MM-DD’) INTO P_DATE FROM DUAL;
ELSE P_DATE:= TO_CHAR (TO_DATE (P_DATE, ‘YYYY-MM-DD’) + 1,’YYYY-MM-DD’); — Increments the day by 1
END CASE;

Aggregation

The aggregation component specifies whether to aggregate the results. Since BI will do this automatically, aggregation and totals are set to 0 (no). An example is:

<aggregate_results>0</aggregate_results> <calculate_totals>0</calculate_totals>

Field Requests

This section may be passed as a parameter and it lists the various data fields to be included in the extract. An example is below:

<day_quota_field>max_available</day_quota_field>
<time_slot_quota_field>max_available</time_slot_quota_field>
<time_slot_quota_field>quota</time_slot_quota_field>
<category_quota_field>used</category_quota_field>
<category_quota_field>used_quota_percent</category_quota_field>
<work_zone_quota_field>status</work_zone_quota_field>

Closing

The Closing section is static text as shown below:

</urn:get_quota_data>
</soapenv:Body>
</soapenv:Envelope>

Calling the SOAP Request

The APEX_WEB_SERVICE package is used to populate a request header and issue the SOAP request. The header requests that the web service return the contents in a non-compressed text format as shown below:

APEX_WEB_SERVICE.G_REQUEST_HEADERS(1).NAME := ‘Accept-Encoding’;
APEX_WEB_SERVICE.G_REQUEST_HEADERS(1).VALUE := ‘identity’;

For each date to be processed the SOAP request envelope is created and issued as shown below:

F_XML := APEX_WEB_SERVICE.MAKE_REQUEST(
P_URL => F_SOAP_URL
,P_ENVELOPE => F_REQUEST_ENVELOPE
,P_WALLET_PATH => ‘file:wallet location’
,P_WALLET_PWD => ‘wallet password‘ );

Troubleshooting the SOAP Request Call

Common issues are the need for a proxy, the need for a trusted certificate (if using HTTPS), and the need to use the TLS security protocol. Note: This post uses DBCS so the second and third issues have been addressed.

The need for a proxy may be detected when the following error occurs: ORA-12535: TNS:operation timed out. Adding the optional p_proxy_override parameter to the call may correct the issue. An example proxy override is:

www-proxy.us.oracle.com

Parsing the SOAP Response

For each date to be processed the SOAP response envelope is parsed to obtain the individual rows and columns.

The hierarchy levels of the capacity API are listed below:

Bucket > Day > Time Slot > Category > Work Zone

Each occurrence of every hierarchical level is parsed to determine attribute names and values. Both the name and the value are then used to populate a column in the staging table.

When a hierarchical level is completed and no occurrences of a lower level exist, a row is inserted into the BICS staging table.

Below is an example XML response element for one bucket.

<bucket>
<bucket_id>TEST Bucket ID</bucket_id>
<name>TEST Bucket Name</name>
<day>
<date>2016-12-21</date>
<time_slot>
<label>7-10</label>
<quota_percent>100</quota_percent>
<quota>2520</quota>
<max_available>2520</max_available>
<used_quota_percent>0</used_quota_percent>
<category>
<label>TEST Category</label>
<quota_percent>100</quota_percent>
<quota>2520</quota>
<max_available>2340</max_available>
<used_quota_percent>0</used_quota_percent>
</category>
</time_slot>
<time_slot>
<label>10-14</label>
<quota_percent>100</quota_percent>
<quota>3600</quota>
<max_available>3600</max_available>
<used_quota_percent>0</used_quota_percent>
<category>
<label>TEST Category</label>
<quota_percent>100</quota_percent>
<quota>3600</quota>
<max_available>3360</max_available>
<used_quota_percent>0</used_quota_percent>
</category>
</time_slot>
<time_slot>
<label>14-17</label>
<quota_percent>100</quota_percent>
<quota>2220</quota>
<max_available>2220</max_available>
<used_quota_percent>0</used_quota_percent>
<category>
<label>TEST Category</label>
<quota_percent>100</quota_percent>
<quota>2220</quota>
<max_available>2040</max_available>
<used_quota_percent>0</used_quota_percent>
</category>
</time_slot>
</day>
</bucket>

The processing of the bucket element is as follows:

Occurrences 1 and 2 of the bucket level are parsed to return attribute names of bucket_id and name. The bucket_id attribute is used as-is and the name attribute is prefixed with “bucket_” to find the corresponding column_ids in the staging view. The corresponding columns in the staging table, C03 and C04, are then populated.

Occurrence 3 of the bucket level returns the day level element tag. Processing then continues at the day level.

Occurrence 1 of the day level returns the attribute name of date. The attribute name is prefixed with “day_” to find the corresponding column_id in the staging view. The corresponding column in the staging table, C05, is then populated with the value ‘2016-12-21’.

Occurrence 2 of the day level returns the first of three time_slot level element tags. Processing for each continues at the time-slot level. Each time_slot element contains 5 attribute occurrences followed by a category level element tag.

Each category level contains 5 attribute occurrences. Note: there is no occurrence of a work_zone level element tag in the category level. Thus after each category level element is processed, a row is written to the staging table.

The end result is that 3 rows are written to the staging table for this bucket. The table below describes the XML to row mapping for the first row.

Attribute Name	Attribute Value	View Column Name	Table Column Name
bucket_id	TEST Bucket ID	BUCKET_ID	C03
name	TEST Bucket Name	BUCKET_NAME	C04
day	2016-12-21	DAY_DATE	C05
label	7-10	TIME_SLOT_LABEL	C18
quota_percent	100	TIME_SLOT_QUOTA_PERCENT	C19
quota	2520	TIME_SLOT_QUOTA	C21
max_available	2520	TIME_SLOT_MAX_AVAILABLE	C26
used_quota_percent	0	TIME_SLOT_USED_QUOTA_PERCENT	C29
label	TEST Category	CAT_LABEL	C32
quota_percent	100	CAT_QUOTA_PERCENT	C33
quota	2520	CAT_QUOTA	C35
max_available	2340	CAT_MAX_AVAILABLE	C42
used_quota_percent	0	CAT_USED_QUOTA_PERCENT	C44

In PL/SQL, the processing is accomplished using the LOOP command. There is a loop for each hierarchical level. Loops end when no results are returned for a parse statement.

XPATH statements are used for parsing. Additional information regarding XPATH statements may be found in the References section. Examples are below:

Statement	Returns
/bucket[5]	The entire fifth bucket element in the response. If no results then all buckets have been processed.
/bucket/*[1]	The first bucket attribute or element name.
/bucket/*[2]/text()	The second bucket attribute value.
/bucket/day/*[6]	The sixth day attribute or element name.
/bucket/day[1]/*[6]/text()	The sixth day attribute value.
/bucket/day/time_slot[2]/*[4]	The fourth attribute or element name of the second time_slot.

Scheduling the Procedure

The procedure may be scheduled to run periodically through the use of an Oracle Scheduler job. A link to the Scheduler documentation may be found in the References section.

A job is created using the CREATE_JOB procedure by specifying a job name, type, action and a schedule. Setting the enabled argument to TRUE enables the job to automatically run according to its schedule as soon as you create it.

An example of a SQL statement to create a job is below:

BEGIN
DBMS_SCHEDULER.CREATE_JOB (
JOB_NAME        => ‘OFSC_SOAP_QUOTA_EXTRACT’,
JOB_TYPE        => ‘STORED_PROCEDURE’,
ENABLED          => TRUE,
JOB_ACTION      => ‘BICS_OFSC_SOAP_INTEGRATION’,
START_DATE      => ’21-DEC-16 10.00.00 PM Australia/Sydney’,
REPEAT_INTERVAL => ‘FREQ=HOURLY; INTERVAL=24’ — this will run the job every 24 hours
);
END;
/

Note: If using the BICS Schema Service database, the package name is CLOUD_SCHEDULER rather than DBMS_SCHEDULER.

The job log and status may be queried using the *_SCHEDULER_JOBS views. Examples are below:

SELECT JOB_NAME, STATE, NEXT_RUN_DATE from USER_SCHEDULER_JOBS;
SELECT LOG_DATE, JOB_NAME, STATUS from USER_SCHEDULER_JOB_LOG;

Summary

This post detailed a method of extracting and loading data from Oracle Field Service Cloud (OFSC) into the Oracle Business Intelligence Cloud Service (BICS).

The post used SOAP web services to extract the XML-formatted data responses. It used a PL/SQL Stored Procedure to wrap the SOAP extract, XML parsing commands, and database table operations. It loaded a BICS staging table and a staging view which can be transformed into star-schema object(s) for use in modeling.

Finally, an example of a database job was provided that executes the Stored Procedure on a scheduled basis.

For more BICS and BI best practices, tips, tricks, and guidance that the A-Team members gain from real-world experiences working with customers and partners, visit Oracle A-Team Chronicles for BICS.