Introduction

This is the second part of a two part article which demonstrates how to upload data in near-real time from an on-premise oracle database to Oracle Storage Cloud Service.

In the previous article of this series, we demonstrated Oracle GoldenGate functionality to write to a flat file using Apache Flume File Roll Sink. If you would like to read the first part in this article series please visit Oracle GoldenGate : Apply to Apache Flume File Roll Sink

In this article we will demonstrate using the cURL command which will upload the flat file to Oracle Storage Cloud Service.

We used the Oracle Big Data Lite Virtual Machine as the test bed for this article. The VM image is available for download on the Oracle Technology Network website.

Main Article

There are various tools available to access Oracle Storage Cloud Service. According to Best Practices – Data movement between Oracle Storage Cloud Service and HDFS , cURL REST interface is appropriate for this requirement.

REST API

REST API is used to manage containers and objects in the Oracle Storage Cloud Service instance. Anyone can access the REST API from any application or programming platform that understands the Hypertext Transfer Protocol (HTTP) and has Internet connectivity.

cURL is one of the tools used to access the REST interface. cURL is an open source tool used for transferring data which supports various protocols including HTTP and HTTPS. cURL is typically available by default on most UNIX-like hosts. For information about downloading and installing cURL, see Quick Start.

Oracle Storage Cloud Service ( OSCS )

Oracle Storage Cloud Service enables applications to store and manage contents in the cloud. Stored objects can be retrieved directly by external clients or by applications running within Oracle Cloud (For example: Big Data Preparation Cloud Service).

A container is a storage compartment that provides a way to organize the data stored in Oracle Storage Cloud Service. Containers are similar to directories, but with a key distinction; unlike directories, containers cannot be nested.

Prerequisites

First, we need access to the Oracle Storage Cloud Service and information about the Oracle Cloud user name, password, and identity domain.

Requesting an Authentication Token

Oracle Storage Cloud Service requires authentication for any operation against the service instance. Authentication is performed by using an authentication token. Authentication tokens are requested from the service by authenticating user credentials with the service. All provisioned authentication tokens are temporary and will expire in 30 minutes. We will include a current authentication token with every request to Oracle Storage Cloud Service.

Request an authentication token by running the following cURL command:

curl -v -s -X GET -H ‘X-Storage-User: <my identity domain>:<Oracle Cloud user name>’ -H ‘X-Storage-Pass: <Oracle Cloud user password>’ https://<myIdentityDomain>.storage.oraclecloud.com/auth/v1.0

We ran the above cURL command. The following is the output of this command, with certain key lines highlighted. Note that if the request includes the correct credentials, it returns the HTTP/1.1 200 OK response.

From the output of the command we just ran, note the following:

– The value of the X-Storage-Url header.

This value is the REST endpoint URL of the service. This URL value will be used in the next step to create the container.

-The value of the X-Auth-Token header.

This value is the authentication token, which will be used in the next step to create the container. Note that the authentication token expires after 30 minutes, after the token expires you should request a fresh token.

Creating A Container

Run the following cURL command to create a new container:

curl -v -s -X PUT -H “X-Auth-Token: <Authentication Token ID>” https://storage.oraclecloud.com/v1/Storage-myIdentityDomain/myFirstContainer

– Replace the value of the X-Auth-Token header with the authentication token that you obtained earlier.
– Change https://storage.oraclecloud.com/v1/Storage-myIdentityDomain to the X-Storage-Url header value that you noted while getting an authentication token.
– And change myFirstContainer to the name of the container that you want to create.

Verifying that A Container is created

 Run the following cURL command:

curl -v -s -X GET -H “X-Auth-Token: <Authentication Token ID>” https://storage.oraclecloud.com/v1/Storage-myIdentityDomain/myFirstContainer

If the request is completed successfully, it returns the HTTP/1.1 204 No Content response. This response indicates that there are no objects yet in new container.

In this exercise, as we are not creating a new container. We will use an existing container to upload the file. So we don’t need to verify the container creation .

Uploading an Object

Once Oracle GolgenGate completes writing the records to a file at /u01/ogg-bd/flumeOut directory, the cURL program reads the file present at /u01/ogg-bd/flumeOut directory. Then it uploads the file to create an object in the  container myFirstContainer. Any user with the Service Administrator role or a role that is specified in the X-Container-Write ACL of the container can create an object.

We ran the following cURL command:

curl -v -X PUT -H “X-Auth-Token:  <Authentication Token ID>”-T myfile https://<MyIdentityDomain>.storage.oraclecloud.com/v1/Storage-myIdentityDomain/myFirstContainer/myObject

When running this command we…
– Replaced the value of the X-Auth-Token header with the authentication token that we obtained earlier.
– Changed https://<MyIdentiryDomain>.storage.oraclecloud.com/v1/Storage-myIdentityDomain to the X-Storage-Url header value that we noted while getting an authentication token.
– Changed myFirstContainer to the name of the container that we want to create.
– Changed myfile  to the full path and name of the file that we want to upload
– Changed myObject to the name of the object that we want to create in the container

If the request is completed successfully, it returns the HTTP/1.1 201 Created response, as shown in the following output. We verified the full transfer by comparing “Content-Length” value.

We also verified the proper transfer of the file to Oracle Storage Cloud Service using Big Data Preparation Cloud Service.

Summary

In this article we demonstrated the functionality of REST API which uploads the data from the On Premise Big Data Lite VM to Oracle Storage Cloud Service.  After combining both articles we demonstrated the functionality of moving data on near real-time from the On-premise Oracle database to Oracle Storage Cloud Service using Oracle Golden Gate and REST API.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission