Introduction

This post details a method of extracting and loading data from Oracle Identity Cloud Service (IDCS) into the Oracle Business Intelligence Cloud Service (BICS). It builds upon the A-team post IDCS Audit Event REST API which details the REST API calls used.

One use case for this method is for analyzing trends regarding audit events.

This post uses REST web services to extract JSON-formatted data responses. It also uses the PL/SQL language to wrap the REST extract, JSON parsing commands, and database table operations in a Stored Procedure. It produces a BICS staging table which can then be transformed into star-schema object(s) for use in modeling. The transformation processes and modeling are not discussed in this post.

Finally, an example of a database job is provided that executes the Stored Procedure on a scheduled basis.

The PL/SQL components are for demonstration purposes only and are not intended for enterprise production use. Additional detailed information, including the complete text of the PL/SQL procedure described, is included in the References section at the end of this post.

Rationale for Using PL/SQL

PL/SQL is the only procedural tool that runs on the BICS / Database Schema Service platform. Other wrapping methods e.g. Java, ETL tools, etc. require a platform outside of BICS to run on.

PL/SQL may also be used in a DBaaS (Database as a Service) that is connected to BICS.

PL/SQL can utilize native SQL commands to operate on the BICS tables. Other methods require the use of the BICS REST API.

Note: PL/SQL is a very good at showcasing functionality. However, it tends to become prohibitively resource intensive when deploying in an enterprise production environment. For the best enterprise deployment, an ETL tool such as Oracle Data Integrator (ODI) should be used to meet these requirements and more:

* Security

* Logging and Error Handling

* Parallel Processing – Performance

* Scheduling

* Code Re-usability and Maintenance

Using Oracle Database as a Service

Determining Security Protocol Requirements

If the web service requires a security protocol, key exchange or cypher not supported by the default BICS Schema Database Service, another Oracle Database Cloud Service (DBaaS) may be used.

Note: For the most consistent response, specify a database version of 11.2.0.4.10 or greater, or any version of 12c. If the database is not at the required version, PL/SQL may throw the following error: ORA-29259: end-of-input reached

To detect what protocol a web service uses, open the IDCS Login page in a browser, click the lock icon, and navigate to the relevant security section. A Chrome example from an IDCS Login page is below:

Preparing the DBaaS

If DBaaS is used, the following steps need to be performed.

Creating the BICS User

Create a BICS user in the database. The use of the Job privilege is discussed later in the post. Example SQL statements are below:

 — USER SQL
CREATE USER “BICS_USER” IDENTIFIED BY password
DEFAULT TABLESPACE “USERS”
TEMPORARY TABLESPACE “TEMP”
ACCOUNT UNLOCK;
— QUOTAS
ALTER USER “BICS_USER” QUOTA UNLIMITED ON USERS;
— ROLES
ALTER USER “BICS_USER” DEFAULT ROLE “CONNECT”,”RESOURCE”;
— SYSTEM PRIVILEGES
GRANT CREATE VIEW TO “BICS_USER”;
GRANT CREATE ANY JOB TO “BICS_USER”;

Managing Trusted Certificates

Create an entry in a new or existing Oracle database wallet for the trusted public certificate used to secure connections to the web service via the Internet. A link to the Oracle Wallet Manager documentation is included in the References section. Note the location and password of the wallet as they are used to issue the REST request.

The need for a trusted certificate is detected when the following error occurs: ORA-29024: Certificate validation failure.

An example certificate path found using Chrome browser is shown below. Both of these trusted certificates need to be in the Oracle wallet.

Granting Network Access

This post uses the UTL_HTTP package which requires the user to have permission to access web services via an Access Control List (ACL).

The need for an ACL privilege is detected when the following error occurs: ORA-24247: network access denied by access control list (ACL).

Grant the BICS_USER authority to connect to the network access control list (ACL). To determine your unique network ACL name run the following:

SELECT * FROM DBA_NETWORK_ACLS;

Using the network name from above run the following:

BEGIN
DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(acl   => ‘NETWORK_ACL_YourUniqueSuffix’
principal   => ‘BICS_USER’,
is_grant    => true,
privilege   => ‘connect’);
END;
/

Preparing the Database Schema

A staging table needs to be created prior to compiling the PL/SQL stored procedure.

This post uses a staging table named AUDIT_EVENT. The columns are those chosen from the REST API for Oracle Identity Cloud Service. A link to the document may be found in the References section. This post uses the following columns:

ACTOR_DISPLAY_NAME
ACTOR_ID
ACTOR_NAME
ACTOR_TYPE
ADMIN_REF_RESOURCE_NAME
ADMIN_RESOURCE_NAME
EC_ID
EVENT_ID
ID
MESSAGE
SSO_COMMENTS
SSO_PROTECTED_RESOURCE
SSO_USER_AGENT
TIMESTAMP

The SQL used to create this table may be viewed here.

Using API Testing Tools

The REST requests should be developed in API testing tools such as SoapUI and Postman. The JSON expressions for parsing should be developed and tested in a JSON expression testing tool such as CuriousConcept. Links to these tools are provided in the References section.

Note: API testing tools such as SoapUI, CuriousConcept, Postman, and so on are third-party tools for using SOAP and REST services. Oracle does not provide support for these tools or recommend a particular tool for its APIs. You can select the tool based on your requirements. As a starting point and for some examples refer to the A-Team post IDCS OAuth 2.0 and REST API.

Preparing and Calling the IDCS REST Service

This post uses the AuditEvents and Token methods of the IDCS REST API 

Preparing the Token Request

IDCS uses the OAuth 2.0 framework for authorization. This requires an access token to be requested and provided via the Token method of the API.

Before preparing the REST request, a Web Application needs to be created in IDCS. This administrative function is not covered in this post. You will need the Client ID and the Client Secret generated with the web application.

You must encode the Client ID and Client Secret when you include it in a request for an access token. A Base64 encoding tool such as Base64 may be used to perform this step. Place the Client ID and Client Secret on the same line and insert a colon between them: clientid:clientsecret and then encode the string. An example encoded result is

Y2xpZW50aWQ6Y2xpZW50c2VjcmV0

You will need the wallet path and password discussed in the Preparing the DBaaS section above. An example path from a linux server is:

/u01/app/oracle

You will need the URL for the Token method of the URL such as:

https://idcs-hostname/oauth2/v1/token

The APEX_WEB_SERVICE package is used to set the headers and parameters described below.

Two HTTP request headers are needed. The first is a Content-Type header and the second is an Authorization header. The authorization header value is the concatenation of the string ‘Basic ‘ with the Base64 encoded result of the Client ID and the Client Secret as shown below:

v_authorization_token := ‘Y2xpZW50aWQ6Y2xpZW50c2VjcmV0’;
apex_web_service.g_request_headers (1).name := ‘Content-Type’;
apex_web_service.g_request_headers(1).value := ‘application/x-www-form-urlencoded; charset=UTF-8’;
apex_web_service.g_request_headers(2).name := ‘Authorization’;
apex_web_service.g_request_headers(2).value := ‘Basic ‘||v_authorization_token ;

The parameter method is set to POST and two HTTP request parameters are needed. The first is a grant_type and the second is a scope as shown below:

p_http_method => ‘POST’,
p_parm_name => apex_util.string_to_table(‘grant_type:scope’),
p_parm_value => apex_util.string_to_table(‘client_credentials~urn:opc:idm:__myscopes__’,’~’)

Note: The urn:opc:idm:__myscopes__ in the scope parameter value is used as a tag by Oracle Identity Cloud Service clients requesting access tokens from the OAuth authorization server. Access tokens are returned that contain all applicable Oracle Identity Cloud Service scopes based on the privileges represented by the Oracle Identity Cloud Service administrator roles granted to the requesting client.

Calling the Token Request

The APEX_WEB_SERVICE package is used to call the request and store the result in a CLOB variable as shown below:

l_ws_response_clob := apex_web_service.make_rest_request (
p_url => l_ws_url,
p_http_method => ‘POST’,
p_parm_name => apex_util.string_to_table(‘grant_type:scope’),
p_parm_value => apex_util.string_to_table (‘client_credentials~urn:opc:idm:__myscopes__’,’~’)
,p_wallet_path => ‘file:/u01/app/oracle‘
,p_wallet_pwd => ‘password‘
);

The result of the call is shown below with a partial token. The token is actually over 2,000 characters long.

{“access_token”:”eyJ4NXQjUzI1NiI6Ijg1a3E1M… “, “token_type”:”Bearer”,”expires_in”:3600}

Note: The response includes the expires_in:3600 parameter. This means that your token is no longer valid after one hour from the time that you generate it.

Parsing the Token Response

The APEX_JSON package is used to parse the token response and store the result in a VARCHAR variable as shown below. Additional information about this package is included as a link in the References section.

apex_json.parse(l_ws_response_clob);
f_idcs_token := apex_json.get_varchar2(p_path => ‘access_token’);

The result of the parse is just the token itself which is used to prepare the Audit Events request.

Preparing the Audit Events Request

The Audit Events request is prepared two or more times. Once to get a first response containing one event that has a field holding the total number of events. Then subsequent requests are made to retrieve all of the events.

IDCS has a limit of how many events are returned for each request. This post uses 500 as a chunk size value which may be modified. Check with the web services administrator for the maximum number of events per request. Also ensure that the number of events inserted into the BICS table equals the total number found in the initial response.

The number of subsequent requests needed is calculated as the total number of events divided by the chunk size, rounded up to the nearest integer. For example 614 events divided by 500 would result in two subsequent requests needed.

The UTL_HTTP package is used instead of the APEX_WEB_SERVICE package to avoid a limitation of 1,024 characters on the length of a header value. The access token is used in a header value and is over 2,000 characters. The error received with the APEX_WEB_SERVICE call is: ORA-06502: PL/SQL: numeric or value error: character string buffer too small.

Preparing All Requests

All requests need to have the following:

The wallet path and password specified. These are specified globally as shown below:

utl_http.set_wallet(‘file:/u01/app/oracle’, ‘password‘); — For Trusted Certificates

Persistent connection support enabled as shown below:

utl_http.set_persistent_conn_support(FALSE, 1); — Set default persistent connections (1)

Begin the request as shown below:

req := utl_http.begin_request(l_ws_url, ‘get’,’http/1.1′);

Note: The result is stored in a variable named req which is of the req type defined in the UTL_HTTP package as shown below:

— A PL/SQL record type that represents a HTTP request
TYPE req IS RECORD (
url VARCHAR2(32767 byte), — Requested URL
method VARCHAR2(64), — Requested method
http_version VARCHAR2(64), — Requested HTTP version
private_hndl PLS_INTEGER — For internal use only
);

The following three HTTP headers set are shown below:

utl_http.set_header(REQ, ‘Content-Type’, ‘application/scim+json’);
utl_http.set_header(REQ, ‘Cache-Control’, ‘no-cache’);
utl_http.set_header(REQ, ‘Authorization’, ‘Bearer ‘ || l_idcs_token); — The received access token

All but the last need persistent connection support as shown below:

utl_http.set_persistent_conn_support(req, TRUE); — Keep Connection Open

Note: The last request does not have the above setting so will default to FALSE and the connection to the service will be closed.

Preparing Individual Requests

Individual requests need to have the following:

The URL set as shown below:

l_ws_url := https://idcs-hostname/admin/v1/AuditEvents?count=1′; — Get first event for total event count

Subsequent URLs are as shown below:

l_ws_url := https://idcs-hostname /admin/v1/AuditEvents?count=500&startIndex=1&sortBy=timestamp;

Note: subsequent requests need the startindex parameter incremented by the chunk size (500).

Calling the Audit Events Request

The Audit Events requests are called using the UTL_HTTP package as shown below:

resp := utl_http.get_response(req);

Note: The result is stored in a variable named resp which is of the resp type defined in the UTL_HTTP package as shown below:

— A PL/SQL record type that represents a HTTP response
TYPE resp IS RECORD (
status_code PLS_INTEGER, — Response status code
reason_phrase VARCHAR2(256), — Response reason phrase
http_version VARCHAR2(64), — Response HTTP version
private_hndl PLS_INTEGER — For internal use only
);

Troubleshooting the REST Request Calls

Common issues are the need for a proxy, the need for an ACL, the need for a trusted certificate (if using HTTPS), and the need to use the correct TLS security protocol. Note: This post uses DBaaS so all but the first issue has been addressed.

The need for a proxy may be detected when the following error occurs: ORA-12535: TNS:operation timed out. Adding the optional p_proxy_override parameter to the call may correct the issue. An example proxy override is:

www-proxy.us.oracle.com

Parsing the Audit Event Responses

The APEX_JSON package is used to parse the responses.

Before parsing begins the staging table is truncated as shown below:

execute immediate ‘truncate table audit_event’;

An example of a response containing just one event is below:

{“schemas”:[“urn:scim:api:messages:2.0:ListResponse”]
,”totalResults”:614
,”Resources”:[
{“eventId”:”sso.authentication.failure”
,”ssoProtectedResource”:”https://idcs-hostname:443/ui/v1/myconsole”
,”actorName”:”user.name@oracle.com”
,”ssoIdentityProvider”:”localIDP”
,”ssoCSR”:”false”
,”ssoUserPostalCode”:”null”
,”ssoUserCity”:”null”
,”reasonValue”:”SSO-1018″
,”ssoUserCountry”:”null”
,”rId”:”0:1:3:2:4″
,”message”:”Authentication failure User not found.”
,”timestamp”:”2016-10-04T09:38:46.336Z”
,”ssoComments”:”Authentication failure User not found.”
,”ssoApplicationHostId”:”idcs-hostname”
,”ssoUserState”:”null”
,”ecId”:”q^Unq0s8000000000″
,”ssoRp”:”IDCS”
,”ssoLocalIp”:”10.196.29.102″
,”serviceName”:”SSO”
,”ssoAuthnLevel”:0
,”actorType”:”User”
,”ssoSessionId”:”null”
,”ssoUserAgent”:”Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36″
,”actorId”:”IDCS”
,”id”:”0a37c7374c494ed080d15c554ae75be8″
,”meta”: {“created”:”2016-10-04T09:38:46.353Z”
,”lastModified”:”2016-10-04T09:38:46.353Z”
,”resourceType”:”AuditEvent”
,”location”:”https://idcs-hostname/admin/v1/AuditEvents/0a37c7374c494ed080d15c554ae75be8″}
,”schemas”:[“urn:ietf:params:scim:schemas:oracle:idcs:AuditEvent”]
,”idcsCreatedBy”: {“value”:”UnAuthenticated”
,”$ref”:”https://idcs-hostname/admin/v1/AuditEvents/UnAuthenticated”}
,”idcsLastModifiedBy”: {“value”:”UnAuthenticated”
,”$ref”:”https://idcs-hostname/admin/v1/AuditEvents/UnAuthenticated”}
}],”startIndex”:1,”itemsPerPage”:1}

Parsing the First Response

The first JSON response of one event is read into a varchar variable as shown below:

utl_http.read_text(resp, l_ws_response_varchar, 32766);

The variable is then parsed as shown below:

apex_json.parse(l_ws_response_varchar);

Note: the above result is implicitly stored in a global package array named g_values. This array contains the JSON members and values.

The value of the JSON member named totalResults is retrieved and stored in a variable as shown below:

v_resultSet := apex_json.get_varchar2(p_path => ‘totalResults’);

This is the total number of events to be retrieved and is all that is wanted from the first response.

Parsing the Subsequent Responses

Subsequent Responses may contain a number of events up to the setting of the chunk size (500 in this post). These responses will need to be stored in a temporary CLOB variable.

The DBMS_LOB package is used to manage the temporary CLOB variable. Additional information about the package may be found in the References section.

This variable is created at the beginning of the parsing and freed at the end of the procedure as shown below:

dbms_lob.createtemporary(l_ws_response_clob, true);
dbms_lob.freetemporary(l_ws_response_clob);

This variable is also trimmed to zero characters at the beginning of each chunk of events using the following:

DBMS_LOB.TRIM (l_ws_response_clob, 0);

The response is read by a LOOP command. Each iteration of the loop reads 32,766 characters of text and appends these to the temporary CLOB variable as shown below:

while not(EOB)
LOOP
BEGIN
utl_http.read_text(resp, l_ws_response_varchar, 32766);
if l_ws_response_varchar is not null and length(l_ws_response_varchar)>0 then
dbms_lob.writeappend(l_ws_response_clob, length(l_ws_response_varchar), l_ws_response_varchar);
end if;
EXCEPTION
WHEN utl_http.end_of_body THEN
EOB := TRUE;
utl_http.end_response(resp);
END;
END LOOP;

The CLOB result is then parsed into the implicit package array of JSON elements and values as shown below. This array contains a number of events equal to or less than the chunk size setting (500).

apex_json.parse(l_ws_response_clob);

Each event in the array is retrieved, has its columns parsed, and is inserted into the BICS staging table as shown below:

for i in 1..v_chunkSize LOOP
v_loadCount := v_loadCount + 1;
IF v_loadCount > v_resultSet THEN NULL;
ELSE
INSERT
INTO AUDIT_EVENT
(
EVENT_ID,
ID,
ACTOR_ID,
ADMIN_REF_RESOURCE_NAME,
ACTOR_NAME,
ACTOR_DISPLAY_NAME,
MESSAGE,
SSO_COMMENTS,
SSO_PROTECTED_RESOURCE,
SSO_USER_AGENT,
TIMESTAMP,
ACTOR_TYPE,
ADMIN_RESOURCE_NAME,
EC_ID
)
VALUES
(
apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].eventId’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].id’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].actorId’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].adminRefResourceName’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].actorName’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].actorDisplayName’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].message’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].ssoComments’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].ssoProtectedResource’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].ssoUserAgent’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].timestamp’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].actorType’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].adminResourceName’)
,apex_json.get_varchar2(p_path => ‘Resources[‘ || i || ‘].ecId’)
);
v_row_count := v_row_count + 1;
END IF;
END LOOP;

After the last chunk of events is processed the procedure terminates.

Scheduling the Procedure

The procedure may be scheduled to run periodically through the use of an Oracle Scheduler job. A link to the Scheduler documentation may be found in the References section.

A job is created using the DBMS_SCHEDULER.CREATE_JOB procedure by specifying a job name, type, action and a schedule. Setting the enabled argument to TRUE enables the job to automatically run according to its schedule as soon as you create it.

An example of a SQL statement to create a job is below:

BEGIN
dbms_scheduler.create_job (
job_name => ‘IDCS_REST_AUDIT_EXTRACT’,
job_type => ‘STORED_PROCEDURE’,
enabled => TRUE,
job_action => ‘BICS_IDCS_REST_INTEGRATION’,
start_date => ’21-DEC-16 10.00.00 PM Australia/Sydney’,
repeat_interval => ‘freq=hourly;interval=24’ — this will run once every 24 hours
);
END;
/

Note: If using the BICS Schema Service database, the package name is CLOUD_SCHEDULER rather than DBMS_SCHEDULER.

The job log and status may be queried using the *_SCHEDULER_JOBS views. Examples are below:

SELECT JOB_NAME, STATE, NEXT_RUN_DATE from USER_SCHEDULER_JOBS;
SELECT LOG_DATE, JOB_NAME, STATUS from USER_SCHEDULER_JOB_LOG;

Summary

This post detailed a method of extracting and loading data from Oracle Identity Cloud Service (IDCS) into the Oracle Business Intelligence Cloud Service (BICS).

The post used REST web services to extract the JSON-formatted data responses. It used a PL/SQL Stored Procedure to wrap the REST extract, JSON parsing commands, and database table operations. It loaded a BICS staging table which can be transformed into star-schema object(s) for use in modeling.

Finally, an example of a database job was provided that executes the Stored Procedure on a scheduled basis.

For more BICS and BI best practices, tips, tricks, and guidance that the A-Team members gain from real-world experiences working with customers and partners, visit Oracle A-Team Chronicles for BICS.

References

Complete Procedure

REST API for Oracle Identity Cloud Service

Scheduling Jobs with Oracle Scheduler

Database PL/SQL Language Reference

APEX_WEB_SERVICE Reference Guide

APEX_JSON Reference Guide

UTL_HTTP Package Reference Guide

Soap API Testing Tool

Curious Concept JOSN Testing Tool

Base64 Decoding and Encoding Testing Tool

Using Oracle Wallet Manager

Oracle Business Intelligence Cloud Service Tasks

DBMS_LOB Reference Guide

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

The objective of this post is to describe how a web app protected by an OAM WebGate can authenticate to the OIG REST APIs. In a previous blog post, I provided detailed steps to do the same thing for the SCIM REST APIs; now in this blog post I will explain how the same approach can be applied to the OIG REST APIs too, with only some minor changes. The reason we can use essentially the same approach for both the OIG REST and SCIM REST APIs is that both use the same OWSM policy (oracle/multi_token_noauth_rest_service_policy) for security. You might use these steps if you were building a custom web interface to OIM, or integrating OIM into a portal (as a custom portlet).

Rather than repeat the steps from that post, I will refer you back to it with the following changes:

1. Firstly, make sure you have applied Bundle Patch 11.1.2.3.161018 (Patch 24326201)
2. Follow the instructions in the Patch 24326201 README to install OIG REST APIs and test them
3. Otherwise follow the steps in my original blog post; however, in “Step 1: Creating the example application”, replace the index.jsp with the alternate version given below.

The index.jsp file needs to be changed to point to the OIG REST API instead of the SCIM REST API. Additionally, the example given was retrieving the /Me SCIM resource which represents the authenticated user; however, that operation has no direct equivalent in the OIG REST API. Instead what we do in this example is parse the SAML assertion to find the username, and then pass that as filter criteria to the /iam/governance/selfservice/api/v1/users endpoint.

<%@page contentType="text/html; charset=UTF-8" %>
<%@ page import="java.io.*" %>
<%@ page import="java.net.*" %>
<%@ page import="java.nio.charset.*" %>
<%@ page import="java.util.*" %>
<%@ page import="java.util.zip.*" %>
<%@ page import="javax.json.*" %>
<%@ page import="javax.json.stream.*" %>
<%@ page import="javax.xml.bind.*" %>
<%@ page import="javax.xml.parsers.*" %>
<%@ page import="org.w3c.dom.*" %>
<%@ page import="org.xml.sax.*" %>

<%!
 
public static String escapeHTML(String s) {
    return s.replaceAll("&","&amp;").replaceAll("<","&lt;").replaceAll(">","&gt;").replaceAll("\"","&quot;");
}
 
public static String gzipBase64(String s) throws Exception {
    byte[] b = s.getBytes(StandardCharsets.UTF_8);
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    GZIPOutputStream gzos = new GZIPOutputStream(baos);
    gzos.write(b);
    gzos.flush();
    gzos.finish();
    return DatatypeConverter.printBase64Binary(baos.toByteArray());
}

public static String getUserName(String samlAssertion) throws Exception {
    StringReader sr = new StringReader(samlAssertion);
    InputSource is = new InputSource(sr);
    DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
    dbf.setNamespaceAware(true);
    DocumentBuilder db = dbf.newDocumentBuilder();
    Document doc = db.parse(is);
    org.w3c.dom.Element root = doc.getDocumentElement();
    NodeList nl = root.getElementsByTagNameNS("urn:oasis:names:tc:SAML:2.0:assertion", "NameID");
    String userName = ((org.w3c.dom.Element)nl.item(0)).getAttribute("SPProvidedID");
    return userName;
}
 
public static JsonObject getMyProfile(String token, String userName) throws Exception {
    URL url = new URL("https://OIMHOST:14001/iam/governance/selfservice/api/v1/users?q=User::Login eq " + userName);
    HttpURLConnection conn = (HttpURLConnection)url.openConnection();
    conn.setRequestProperty("Authorization","oit " + token);
    StringBuilder sb = new StringBuilder();
    JsonReader rdr = Json.createReader(conn.getInputStream());
        try {
        return rdr.readObject();
    } finally {
        rdr.close();
    }
}
 
public static String prettyPrint(JsonObject obj) {
    Map<String, Object> cfg = new HashMap<String,Object>(1);
    cfg.put(JsonGenerator.PRETTY_PRINTING, true);
    StringWriter sw = new StringWriter();
    JsonWriterFactory wf = Json.createWriterFactory(cfg);
    JsonWriter jw = wf.createWriter(sw);
    jw.writeObject(obj);
    jw.close();
    return sw.toString();
}
 
public static String htmlesc(String input) {
    return input.replace("&","&amp;").replace("<","&lt;").replace(">","&gt;").replace("\"","&quot;");
}
     
%>
<%
    String samlAssertion = request.getHeader("OAM_IDENTITY_ASSERTION");
    String token = gzipBase64(samlAssertion);
    String userName = getUserName(samlAssertion);
    JsonObject profile = getMyProfile(token, userName);
 
%>
<html>
<head>
<title>Show My Profile</title>
</head>
<body>
<h1>Show My Profile</h1>
Find below your user profile.
<pre><%=htmlesc(prettyPrint(profile))%></pre>
</body>
</html>

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

The objective of this post is to show the differences and similarities of the two REST APIs offered by OIM – the SCIM REST API and the OIG REST API.

OIM Java APIs have been available in OIM for many versions now (since OIM 9.x or earlier), although each successive version has added new Java APIs to provide access to new functionality. The SCIM REST web services were newly introduced in 11.1.2.3. My previous blog post contains some discussions of the pros and cons of each API and when to use each one.

In Bundle Patch 11.1.2.3.161018 (Patch 24326201), a third option has been introduced – the OIG REST API. The OIG REST API is not automatically installed with the bundle patch but is a separate installation procedure (see the bundle patch README for details). It is documented here.

Note that while the two REST APIs differ both in their capabilities and in the syntax used to invoke those capabilities, one area of commonality is security. Both APIs delegate their security to OWSM and use the same OWSM policy, so they are secured identically. On that topic please see, in addition to this post, my colleague Pulkit’s post on securing OIM REST APIs.

SCIM REST and OIG REST APIs compared

The table below provides a side-by-side comparison of the SCIM REST API with the OIG REST API:

Detailed comparison of capabilities

As you can see from the above, there are some capabilities both APIs have in common and there are also some capabilities unique to one or the other. However, overall, it is clear that the OIG REST API offers a richer selection of capabilities than the SCIM REST API does.

Note that the JSON Web Token (JWT) issuance endpoints strictly speaking form part of the OIG REST API, but the tokens they generate are equally usable with the SCIM REST API.

Obviously, if you need the additional functionality that the OIG REST API alone provides, then the OIG REST API is the answer. If you don’t need that additional functionality, then the standardisation and cloud interoperability that the SCIM REST API provides could be a good reason to choose the SCIM REST API.

Note that it is possible to use both APIs in the one application. However, if you choose to do that, you should make sure you are choosing which API to use in each case based on rational criteria rather than random whims. For example, you might decide that you want to use the SCIM REST API wherever possible, and only use the OIG REST API when you need the additional functionality which the OIG REST API alone provides.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

Introduction

This post details a method of extracting and loading data from Oracle Field Service Cloud (OFSC) into the Oracle Business Intelligence Cloud Service (BICS) using RESTful services. It is a companion to the A-Team post Loading Data from Oracle Field Service Cloud into Oracle BI Cloud Service using SOAP . Both this post and the SOAP post offer methods to compliment the standard OFSC Daily Extract described in Oracle Field Service Cloud Daily Extract Description.

One case for using this method is analyzing trends regarding OFSC events.

This post uses RESTful web services to extract JSON-formatted data responses. It also uses the PL/SQL language to call the web services, parse the JSON responses, and perform database table operations in a Stored Procedure. It produces a BICS staging table which can then be transformed into star-schema object(s) for use in modeling. The transformation processes and modeling are not discussed in this post.

Finally, an example of a database job is provided that executes the Stored Procedure on a scheduled basis.

The PL/SQL components are for demonstration purposes only and are not intended for enterprise production use. Additional detailed information, including the complete text of the PL/SQL procedure described, is included in the References section at the end of this post.

Rationale for Using PL/SQL

PL/SQL is the only procedural tool that runs on the BICS / Database Schema Service platform. Other wrapping methods e.g. Java, ETL tools, etc. require a platform outside of BICS to run on.

PL/SQL may also be used in a DBaaS (Database as a Service) that is connected to BICS.

PL/SQL can utilize native SQL commands to operate on the BICS tables. Other methods require the use of the BICS REST API.

Note: PL/SQL is a very good at showcasing functionality. However, it tends to become prohibitively resource intensive when deploying in an enterprise production environment. For the best enterprise deployment, an ETL tool such as Oracle Data Integrator (ODI) should be used to meet these requirements and more:

* Security

* Logging and Error Handling

* Parallel Processing – Performance

* Scheduling

* Code Re-usability and Maintenance

About the OFSC REST API

The document REST API for Oracle Field Service Cloud Service should be used extensively, especially the Authentication, Paginating, and Working with Events sections. Terms described there such as subscription, page, and authorization are used in the remainder of this post.

In order to receive events, a subscription is needed listing the specific events desired. The creation of a subscription returns both a subscription ID and a page number to be used in the REST calls to receive events.

At this time, a page contains 0 to 100 items (events) along with the next page number to use in a subsequent call.

The following is a list of supported events types available from the REST API:

Activity Events
Activity Link Events
Inventory Events
Required Inventory Events
User Events
Resource Events
Resource Preference Events

This post uses the following subset of events from the Activity event type:

activityCreated
activityUpdated
activityStarted
activitySuspended
activityCompleted
activityNotDone
activityCanceled
activityDeleted
activityDelayed
activityReopened
activityPreworkCreated
activityMoved

The process described in this post can be modified slightly for each different event type. Note: the columns returned for each event type differ slightly and require modifications to the staging table and parsing section of the procedure.

Using Oracle Database as a Service

This post uses the new native support for JSON offered by the Oracle 12c database. Additional information about these new features may be found in the document JSON in Oracle Database.

These features provide a solution that overcomes a current limitation in the APEX_JSON package. The maximum length of JSON values in that package is limited to 32K characters. Some of the field values in OFSC events exceed this length.

Preparing the DBaaS Wallet

Create an entry in a new or existing Oracle database wallet for the trusted public certificates used to secure connections to the web service via the Internet. A link to the Oracle Wallet Manager documentation is included in the References section. Note the location and password of the wallet as they are used to issue the REST request.

The need for a trusted certificate is detected when the following error occurs: ORA-29024: Certificate validation failure.

An example certificate path found using Chrome browser is shown below. Both of these trusted certificates need to be in the Oracle wallet.

Creating a BICS User in the Database

The complete SQL used to prepare the DBaaS may be viewed here.

Example SQL statements are below:

CREATE USER “BICS_USER” IDENTIFIED BY password
DEFAULT TABLESPACE “USERS”
TEMPORARY TABLESPACE “TEMP”
ACCOUNT UNLOCK;
— QUOTAS
ALTER USER “BICS_USER” QUOTA UNLIMITED ON USERS;
— ROLES
ALTER USER “BICS_USER” DEFAULT ROLE “CONNECT”,”RESOURCE”;
— SYSTEM PRIVILEGES
GRANT CREATE VIEW TO “BICS_USER”;
GRANT CREATE ANY JOB TO “BICS_USER”;

Creating Database Schema Objects

Three tables need to be created prior to compiling the PL/SQL stored procedure. These tables are:

*     A staging table to hold OFSC Event data

*     A subscription table to hold subscription information.

*     A JSON table to hold the JSON responses from the REST calls

The staging table, named OFSC_EVENT_ACTIVITY, has columns described in the OFSC REST API for the Activity event type. These columns are:

PAGE_NUMBER — for the page number the event was extracted from
ITEM_NUMBER — for the item number within the page of the event
EVENT_TYPE
EVENT_TIME
EVENT_USER
ACTIVITY_ID
RESOURCE_ID
SCHEDULE_DATE
APPT_NUMBER
CUSTOMER_NUMBER
ACTIVITY_CHANGES — To store all of the individual changes made to the activity

The subscription table, named OFSC_SUBSCRIPTION_PAGE, has the following columns:

SUBSCRIPTION_ID     — for the supported event types
NEXT_PAGE                — for the next page to be extracted in an incremental load
LAST_UPDATE            — for the date of the last extract
SUPPORTED_EVENT — for the logical name for the subscription event types
FIRST_PAGE               — for the first page to be extracted in a full load

The JSON table, named OFSC_JSON_TMP, has the following columns:

PAGE_NUMBER — for the page number extracted
JSON_CLOB       — for the JSON response received for each page

Using API Testing Tools

The REST requests should be developed in API testing tools such as cURL and Postman. The JSON expressions for parsing should be developed and tested in a JSON expression testing tool such as CuriousConcept. Links to these tools are provided in the References section.

Note: API testing tools such as SoapUI, CuriousConcept, Postman, and so on are third-party tools for using SOAP and REST services. Oracle does not provide support for these tools or recommend a particular tool for its APIs. You can select the tool based on your requirements.

Subscribing to Receive Events

Create subscriptions prior to receiving events. A subscription specifies the types of events that you want to receive. Multiple subscriptions are recommended. For use with the method in this post, a subscription should only contain events that have the same response fields.

The OFSC REST API document describes how to subscribe using a cURL command. Postman can also easily be used. Either tool will provide a response as shown below:

{
“subscriptionId”: “a0fd97e62abca26a79173c974d1e9c19f46a254a”,
“nextPage”: “160425-457,0”,
“links”: [ … omitted for brevity ]
}.

Note: The default next page is for events after the subscription is created. Ask the system administrator for a starting page number if a past date is required.

Use SQL*Plus or SQL Developer and insert a row for each subscription into the OFSC_SUBSCRIPTION_PAGE table.

Below is an example insert statement for the subscription above:

INSERT INTO OFSC_SUBSCRIPTION_PAGE
(
SUBSCRIPTION_ID,
NEXT_PAGE,
LAST_UPDATE,
SUPPORTED_EVENT,
FIRST_PAGE
)
VALUES
(
‘a0fd97e62abca26a79173c974d1e9c19f46a254a’,
‘160425-457,0’,
sysdate,
‘Required Inventory’,
‘160425-457,0’
);

Preparing and Calling the OFSC RESTful Service

This post uses the events method of the OFSC REST API.

This method requires the Basic framework for authorization and mandates a base64 encoded value for the following information: user-login “@” instance-id “:” user-password

An example encoded result is:

dXNlci1sb2dpbkBpbnN0YW5jZS1pZDp1c2VyLXBhc3N3b3Jk

The authorization header value is the concatenation of the string ‘Basic’ with the base64 encoded result discussed above. The APEX_WEB_SERVICE package is used to set the header as shown below:

v_authorization_token := ‘ dXNlci1sb2dpbkBpbnN0YW5jZS1pZDp1c2VyLXBhc3N3b3Jk’;
apex_web_service.g_request_headers(1).name  := ‘Authorization’;
apex_web_service.g_request_headers(1).value := ‘Basic ‘||v_authorization_token;

The wallet path and password discussed in the Preparing the DBaaS Wallet section are also required. An example path from a Linux server is:

/u01/app/oracle

Calling the Events Request

The events request is called for each page available for each subscription stored in the OFSC_SUBSCRIPTION_PAGE table using a cursor loop as shown below:

For C1_Ofsc_Subscription_Page_Rec In C1_Ofsc_Subscription_Page
Loop
V_Subscription_Id := C1_Ofsc_Subscription_Page_Rec.Subscription_Id;
Case When P_Run_Type = ‘Full’ Then
V_Next_Page := C1_Ofsc_Subscription_Page_Rec.First_Page;
Else
V_Next_Page := C1_Ofsc_Subscription_Page_Rec.Next_Page;
End Case; … End Loop;

The URL is modified for each call. The subscription_id and the starting page are from the table.

For the first call only, if the parameter / variable p_run_type is equal to ‘Full’, the staging table is truncated and the page value is populated from the FIRST_PAGE column in the OFSC_SUBSCRIPTION_PAGE table. Otherwise, the staging table is not truncated and the page value is populated from the NEXT_PAGE column.

Subsequent page values come from parsing the nextPage value in the responses.

An example command to create the URL from the example subscription above is:

f_ws_url := v_base_url||’/events?subscriptionId=’ ||v_subscription_id|| chr(38)||’page=’ ||v_next_page;

The example URL result is:

https://ofsc-hostname/rest/ofscCore/v1/events?subscriptionId=a0fd97e62abca26a79173c974d1e9c19f46a254a&page=160425-457,0

An example call using the URL is below:

f_ws_response_clob := apex_web_service.make_rest_request (
p_url => f_ws_url
,p_http_method => ‘GET’
,p_wallet_path => ‘file:/u01/app/oracle’
,p_wallet_pwd => ‘wallet-password‘ );

Storing the Event Responses

Each response (page) is processed using a while loop as shown below:

While V_More_Pages
Loop
Extract_Page;
End Loop;

Each page is parsed to obtain the event type of the first item. A null (empty) event type signals an empty page and the end of the data available. An example parse to obtain the event type of the first item is below. Note: for usage of the JSON_Value function below see JSON in Oracle Database.

select  json_value (f_ws_response_clob, ‘$.items[0].eventType’ ) into f_event_type from  dual;

If there is data in the page, the requested page number and the response clob are inserted into the OFSC_JSON_TMP table and the response is parsed to obtain the next page number for the next call as shown below:

f_json_tmp_rec.page_number := v_next_page; — this is the requested page number
f_json_tmp_rec.json_clob := f_ws_response_clob;
insert into ofsc_json_tmp values f_json_tmp_rec;
select json_value (f_ws_response_clob, ‘$.nextPage’ ) into v_next_page from dual;

Parsing and Loading the Events Responses

Each response row stored in the OFSC_JSON_TMP table is retrieved and processed via a cursor loop statement as shown below:

for c1_ofsc_json_tmp_rec in c1_ofsc_json_tmp
loop
process_ofsc_json_page (c1_ofsc_json_tmp_rec.page_number);
end loop;

An example response is below with only the first item shown:

{
“found”: true,
“nextPage”: “170110-13,0”,
“items”: [
{
“eventType”: “activityUpdated”,
“time”: “2017-01-04 12:49:51”,
“user”: “soap”,
“activityDetails”: {
“activityId”: 1297,
“resourceId”: “test-resource-id“,
“resourceInternalId”: 2505,
“date”: “2017-01-25”,
“apptNumber”: “82994469003”,
“customerNumber”: “12797495”
},
“activityChanges”: {
“A_LastMessageStatus”: “SuccessFlag – Fail – General Exception: Failed to update FS WorkOrder details. Reason: no rows updated for: order_id = 82994469003 service_order_id = NULL”
}
}
],
“links”: [
…
]
}

Each item (event) is retrieved and processed via a while loop statement as shown below:

while f_more_items loop
process_item (i);
i := i + 1;
end loop;

For each item, a dynamic SQL statement is prepared and submitted to return the columns needed to insert a row into the OFSC_EVENT_ACTIVITY staging table as shown below (the details of creating the dynamic SQL statement have been omitted for brevity):

An example of a dynamically prepared SQL statement is below. Note: for usage of the JSON_Table function below see JSON in Oracle Database.

The execution of the SQL statement and the insert are shown below:

execute immediate f_sql_stmt into ofsc_event_activity_rec;
insert into ofsc_event_activity values ofsc_event_activity_rec;

Verifying the Loaded Data

Use SQL*Plus, SQL Developer, or a similar tool to display the rows loaded into the staging table.

A sample set of rows is shown below:

Troubleshooting the REST Calls

Common issues are the need for a proxy, the need for an ACL, the need for a trusted certificate (if using HTTPS), and the need to use the correct TLS security protocol. Note: This post uses DBaaS so all but the first issue has been addressed.

The need for a proxy may be detected when the following error occurs: ORA-12535: TNS:operation timed out. Adding the optional p_proxy_override parameter to the call may correct the issue. An example proxy override is:

www-proxy.us.oracle.com

Scheduling the Procedure

The procedure may be scheduled to run periodically through the use of an Oracle Scheduler job as described in Scheduling Jobs with Oracle Scheduler.

A job is created using the DBMS_SCHEDULER.CREATE_JOB procedure by specifying a job name, type, action and a schedule. Setting the enabled argument to TRUE enables the job to automatically run according to its schedule as soon as you create it.

An example of a SQL statement to create a job is below:

BEGIN
dbms_scheduler.create_job (
job_name => ‘OFSC_REST_EVENT_EXTRACT’,
job_type => ‘STORED_PROCEDURE’,
enabled => TRUE,
job_action => ‘BICS_OFSC_REST_INTEGRATION’,
start_date => ’12-JAN-17 11.00.00 PM Australia/Sydney’,
repeat_interval => ‘freq=hourly;interval=24’ — this will run once every 24 hours
);
END;
/

Note: If using the BICS Schema Service database, the package name is CLOUD_SCHEDULER rather than DBMS_SCHEDULER.

The job log and status may be queried using the *_SCHEDULER_JOBS views. Examples are below:

SELECT JOB_NAME, STATE, NEXT_RUN_DATE from USER_SCHEDULER_JOBS;
SELECT LOG_DATE, JOB_NAME, STATUS from USER_SCHEDULER_JOB_LOG;

Summary

This post detailed a method of extracting and loading data from Oracle Field Service Cloud (OFSC) into the Oracle Business Intelligence Cloud Service (BICS) using RESTful services.

The method extracted JSON-formatted data responses and used the PL/SQL language to call the web services, parse the JSON responses, and perform database table operations in a Stored Procedure. It also produced a BICS staging table which can then be transformed into star-schema object(s) for use in modeling.

Finally, an example of a database job was provided that executes the Stored Procedure on a scheduled basis.

For more BICS and BI best practices, tips, tricks, and guidance that the A-Team members gain from real-world experiences working with customers and partners, visit Oracle A-Team Chronicles for BICS.

References

Complete Procedure

JSON in Oracle Database

REST API for Oracle Field Service Cloud Service

Scheduling Jobs with Oracle Scheduler

Database PL/SQL Language Reference

APEX_WEB_SERVICE Reference Guide

APEX_JSON Reference Guide

Curious Concept JSON Testing Tool

Postman Testing Tool

Base64 Decoding and Encoding Testing Tool

Using Oracle Wallet Manager

Oracle Business Intelligence Cloud Service Tasks

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission

(or Smart devices, not so smart protocols)I've been working on and off with a customer on a project that involves all sorts of cool buzzwords - iPhone/Android/Blackberry Apps as clients, using REST to invoke Web Services, authenticating via SAML. While...