Continuous integration with Jenkins - Tutorial

Jenkins

This article describes how to build a continuous integration cycle for Java development with the Jenkins continuous integration build server.

Table of Contents

1. Using the Jenkins build server

2. Installation

2.1. Installing of the Jenkins server on Ubuntu

2.3. Using the .WAR file of Jenkins

3. Configure Jenkins

3.1. Configuration the JDK location

3.2. Secure Jenkins

3.3. Generate ssh key for Jenkins user

4. Jenkins management

4.1. Plug-in management

4.2. Restart your Jenkins

5. Support for the Git version control systems

6. Setting up a Jenkins job

7. Prerequisites for build Android apps with Jenkins

8. How to build your Android application with Jenkins

8.1. Android Jenkins build job prerequisites

8.2. Installation of the Android SDK

8.3. Jenkins Plug-ins for installation

8.4. Running an Android build with Jenkins

8.5. Other useful plug-ins

9. Jenkins backup and copying files

9.1. Jenkins backup and copying files

9.2. Managing Jenkins with Git

1. Using the Jenkins build server

Continuous integration is a process in which all development work is integrated as early as possible and the resulting artifacts are automatically created and tested. This process should identify errors as very early in the process.

Jenkins is one open source tool to perform continuous integration and build automation. The basic functionality of Jenkins is to execute a predefined list of steps. The trigger for this execution can be time or event based. For example every 20 minutes or after a new commit in a Git repository.

The list of steps can, for example, include:

• perform a software build with Apache Maven or Gradle

• Run a shell script

• Archive the build result

• Afterwards start the integration tests

Jenkins also monitors the execution of the steps and allows to stop the process if one of the steps fails. Jenkins can also send out notifications about the build success or failure.

Jenkins can be extended by additional plug-ins, e.g., for building and testing Android applications or to support the Git version control system.

2. Installation

Jenkins can be started via the command line or can run in a web application server. Under Linux you can also install Jenkins as a system service.

For most platforms you have native packages, see the Jenkins Homepage.

2.2. Installing of the Jenkins server on Ubuntu

Jenkins provides Debian/Ubuntu packages which install Jenkins and register Jenkins as start service. See the Install Jenkins on Ubuntu description.

Jenkins stores all the settings, logs and build artifacts in its home directory. The default installation directory is/var/lib/jenkins under Ubuntu.

This creates a /etc/init.d/jenkins start script which starts Jenkins automatically at boot time. If you installed Jenkins locally, you find it running under the following URL: http://localhost:8080/

2.3. Using the .WAR file of Jenkins

Download the jenkins.war file from Jenkins Homepage.

You can also start Jenkins directly via the command line with java -jar jenkins*.war. If you start it locally, you find it running under the following URL: http://localhost:8080/

To run it in your Tomcat server, put the .WAR file into the webapps directory. If you start Tomcat, your Jenkins installation will be available under http://localhost:8080/jenkins

Note: If the jenkins.war is deployed in your webapps directory, but cannot be started and the tomcat manager says “FAIL - Application at context path /jenkins could not be started ”, you may need to grant the permissons for JENKINS_HOME.

cd /usr/share/tomcat7

            sudo mkdir .jenkins

            sudo

            chown tomcat7:nogroup .jenkins 

This makes the .jenkins folder writable and Jenkins can use it.

3. Configure Jenkins

3.1. Configuration the JDK location

Before using Jenkins to build Java applications, you need to configure the location or it where your JDK installation is. Select Manage Jenkins and afterwards Configure System.

Enter the correct path to your JDK, Apache Ant and Maven and press the Save button below. Jenkins can also install these for your automatically.

3.2. Secure Jenkins

It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable security flag. The easiest way is to use Jenkins own user database. Create at least the user "Anonymous" with read access. Also create entries for the users you want to add in the next step.

On the login page, select Create an account to create the users you just gave access

Go to Manage Jenkins, Manage and Assign Roles and then Assign Roles to grant the newly created user additional access rights.

Navigate to Manage Roles to define access restrictions in detail. Pattern is a regex value of the job name. The following grants unregistered users read-only access to your build jobs that start with the C-MASTER or M-MASTER prefix and only those.

3.3. Generate ssh key for Jenkins user

If you want to access a private Git repo, for example at Github, you need to generate an ssh key-pair. Create a SSH key with the following command: sudo -u jenkins ssh-keygen.

4. Jenkins management

4.1. Plug-in management

Jenkins can be extended via additional plug-ins with more functionality. You can configure your plug-ins via the Manage Jenkins → Manager Plugins link.

To install plugins in Jenkins select use the Manage Jenkins → Manager Plugins link and search for the plugin you want to install. Select it from the list and select to install it and restart Jenkins.

The following table is a summary of commonly used plug-ins.

Table 1. Jenkins plug-ins

4.2. Restart your Jenkins

You can manually restart Jenkins by adding restart as URL parameter

5. Support for the Git version control systems

Jenkins supports the Git version control system via a plugin. Select the Manage Jenkins → Manager Plugins link. Here you have to install the Git Plugin.

To clone a Git repostory via Jenkins you need to enter the email and user name for your Jenkins system. For this switch into your job directory and run the git config command.

6. Setting up a Jenkins job

The build of a project is handled via jobs in Jenkins. Select New Item from the menu

Afterwards enter a name for the job and select Freestyle Job. Press OK to create a new Job in Jenkins.

The next page allows you to configure your job. If you for example using Git, enter the URL to the Git repository. If the repository is not public, you may also need to configure the credentials.

Specify when and how your build should be triggered. The following example polls the Git repository every 15 min and triggers a build, if something has changed in the repo.

I typically delete the workspace before a build to avoid any side-effect. In the Build section you can add a build step, e.g., a Maven build.

Press Save to finish the job definition. Press Build Now on the job page to validate the job works as expected.

After a while the job should go to green or blue (depending on your configuration), if successful. Click on the job and afterwards on Console Output to see the log file in case of an error or to validate that log looks as expected.

7. Prerequisites for build Android apps with Jenkins

The following descriptions for build Android applications with Jenkins assume that you have already a correct Gradle build file for your Android application. See Android build tutorial.

8. How to build your Android application with Jenkins

8.1. Android Jenkins build job prerequisites

Jenkins supports the automatic building and testing of Android applications. To create a build job on Jenkins you need to have a working build setup. The Android tooling generates automatically a valid Gradle build file.

8.2. Installation of the Android SDK

The Android SDk must be available on the server. If you build server has no display server you need to install the Android SDK from the command line. You also need to configure the location for the Android SKD for Jenkins.

Use android list targets to see what ABIs are installed. If no are listed manual install one.

8.3. Jenkins Plug-ins for installation

You need the follwing plug-ins to build Android application with Jenkins:

• Gradle Plugin

• Git Plugin - if Git is used for the project sources

• Android Emulator Plugin - supporting starting and unlocking an Android emulator and blocks the build until the emulator has started. Required for running tests. A detailed description of this plugin can be found under the following URL: Android Emulator Plugin Jenkins page.

8.4. Running an Android build with Jenkins

To create an Android build job on Jenkins, select New Job, enter a job name and select the Build a free-style software project option.

You configure from where the source should be cloned.

You configure the emulator which should be started. Ensure that you do not select the Show emulator window option, as your build server should not depend on the availability of a display server.

Add a Gradle build step to your Jenkins jobs.

8.5. Other useful plug-ins

The Android Emulator Plugin supports a new job with the Build multi-configuration project option. This option allows you to test multiple emulator configurations at the same time. You can, for example, test different languages, densities, screen resolutions, etc. Typically you have two Jobs, one for a simple build and test run and a multi-configuration project to test the build on different device configurations.

You can combine Android Emulator Plugin with the Amazon-EC2-Plug-in to run the build and theJenk tests on several machines simultaneously.

Another useful plug-in is the Lint plug-in which allows to run your Lint checks via Jenkins. See Lint Plug-in for details.

9. Jenkins backup and copying files

9.1. Jenkins backup and copying files

Jenkins stores all the settings, logs and build artifacts in its home directory, for example, in /var/lib/jenkins under the default install location of Ubuntu.

To create a backup of your Jenkins setup, just copy this directory.

The jobs directory contains the individual jobs configured in the Jenkins install. You can move a job from one Jenkins installation to another by copying the corresponding job directory. You can also copy a job directory to clone a job or rename the directory.

Click reload config button in the Jenkins web user interface to force Jenkins to reload configuration from the disk.

See Adminstration of Jenkins for details.

9.2. Managing Jenkins with Git

Jenkins supports the https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin plug-in which allows you to store every change in a Git repo.

It is also possible to manually maintain the Jenkins configuration in a Git repo.

Enabling Logging on Oracle Identity Manager(OIM) during provisioning and reconciliation operations at logging.xml file

In order to troubleshoot provisioning and reconciliation operations issues enable logging after following below procedures.

configuration file is located atDOMAIN_HOME/config/fmwconfig/servers/OIM_SERVER/logging.xml

1. always please make sure to take backup logging.xml of  before updating the values.


2. chose what kind of logging you need.

Java Level	ODL Message Type:Level
SEVERE.intValue()+100	INCIDENT_ERROR:1
SEVERE	ERROR:1
WARNING	WARNING:1
INFO	NOTIFICATION:1
CONFIG	NOTIFICATION:16
FINE	TRACE:1
FINER	TRACE:16
FINEST	TRACE:32

3. vi  logging.xml 

1. Add the following blocks in the file:

2. <log_handler name='OIMCP.LDAP' level='[LOG_LEVEL]' class='oracle.core.ojdl.logging.ODLHandlerFactory'>
   <property name='logreader:' value='off'/>
        <property name='path' value='[FILE_NAME]'/>
        <property name='format' value='ODL-Text'/>
        <property name='useThreadName' value='true'/>
        <property name='locale' value='en'/>
        <property name='maxFileSize' value='5242880'/>
        <property name='maxLogSize' value='52428800'/>
        <property name='encoding' value='UTF-8'/>
      </log_handler>
   

4. <logger name="ORG.IDENTITYCONNECTORS.LDAP" level="[LOG_LEVEL]" useParentHandlers="false">
        <handler name="OIMCP.LDAP"/>
        <handler name="console-handler"/>
      </logger>
   

5. Replace both occurrences of [LOG_LEVEL] with the ODL message type and level combination that you require.
6. Replace [FILE_NAME] with the full path and name of the log file in which you want log messages to be recorded.

3. Save and close the file.
4. Set the following environment variable to redirect the server logs to a file:

For Microsoft Windows:
set WLS_REDIRECT_LOG=FILENAME 
For UNIX:
export WLS_REDIRECT_LOG=FILENAME 
Replace FILENAME with the location and name of the file to which you want to redirect the output.

5.Restart the application server.

Oracle Data Integrator (ODI) - Frequently Asked Questions (FAQ)

1) What is Oracle Data Integrator (ODI)?
Oracle acquired Sunopsis in 2006 and with it "Sunopsis Data Integrator".

Oracle Data Integrator (ODI) is an E-LT (Extract, Load and Transform) tool used for high-speed data movement between disparate systems.

The latest version, Oracle Data Integrator Enterprise Edition (ODI-EE) brings together "Oracle Data Integrator" and "Oracle Warehouse Builder" as separate components of a single product with a single licence.


2) What is E-LT?E-LT is an innovative approach to extracting, loading and Transforming data. Typically ETL application vendors have relied on costly heavyweight , mid-tier server to perform the transformations required when moving large volumes of data around the enterprise.

ODI delivers unique next-generation, Extract Load and Transform (E-LT) technology that improves performance and reduces data integration costs, even across heterogeneous systems by pushing the processing required down to the typically large and powerful database servers already in place within the enterprise.


3) What components make up Oracle Data Integrator?"Oracle Data Integrator" comprises of:

- Oracle Data Integrator + Topology Manager + Designer + Operator + Agent
- Oracle Data Quality for Data Integrator
- Oracle Data Profiling

4) What is Oracle Data Integration Suite?

Oracle data integration suite is a set of data management applications for building, deploying, and managing enterprise data integration solutions:

• Oracle Data Integrator Enterprise Edition
• Oracle Data Relationship Management
• Oracle Service Bus (limited use)
• Oracle BPEL (limited use)
• Oracle WebLogic Server (limited use)

Additional product options are:

• Oracle Goldengate
• Oracle Data Quality for Oracle Data Integrator (Trillium-based DQ)
• Oracle Data Profiling (Trillium based Data Profiling)
• ODSI (the former Aqualogic Data Services Platform)

5) What systems can ODI extract and load data into?
ODI brings true heterogeneous connectivity out-of-the-box, it can connect natively to Oracle, Sybase, MS SQL Server, MySQL, LDAP, DB2, PostgreSQL, Netezza.

It can also connect to any data source supporting JDBC, its possible even to use the Oracle BI Server as a data source using the jdbc driver that ships with BI Publisher

6) What are Knowledge Modules?
Knowledge Modules form the basis of 'plug-ins' that allow ODI to generate the relevant execution code , across technologies , to perform tasks in one of six areas, the six types of knowledge module consist of:

• Reverse-engineering knowledge modules are used for reading the table and other object metadata from source databases
• Journalizing knowledge modules record the new and changed data within either a single table or view or a consistent set of tables or views
• Loading knowledge modules are used for efficient extraction of data from source databases for loading into a staging area (database-specific bulk unload utilities can be used where available)
• Check knowledge modules are used for detecting errors in source data
• Integration knowledge modules are used for efficiently transforming data from staging area to the target tables, generating the optimized native SQL for the given database
• Service knowledge modules provide the ability to expose data as Web services

ODI ships with many knowledge modules out of the box, these are also extendable, they can modified within the ODI Designer module.

7) How do 'Contexts' work in ODI?
ODI offers a unique design approach through use of Contexts and Logical schemas. Imagine a development team, within the ODI Topology manager a senior developer can define the system architecture, connections, databases, data servers (tables etc) and so forth.

These objects are linked through contexts to 'logical' architecture objects that are then used by other developers to simply create interfaces using these logical objects, at run-time, on specification of a context within which to execute the interfaces, ODI will use the correct physical connections, databases + tables (source + target) linked the logical objects being used in those interfaces as defined within the environment Topology.


8) Does my ODI infrastructure require an Oracle database?
No, the ODI modular repositories (Master + and one of multiple Work repositories) can be installed on any database engine that supports ANSI ISO 89 syntax such as Oracle, Microsoft SQL Server, Sybase AS Enterprise, IBM DB2 UDB, IBM DB2/40.


9) Where can I get more information on ODI?
The OTN Data integration home page : http://www.oracle.com/us/products/middleware/data-integration/index.html

10) Does ODI support web services?
Yes, ODI is 'SOA' enabled and its web services can be used in 3 ways:

• The Oracle Data Integrator Public Web Service, that lets you execute a scenario (a published package) from a web service call
• Data Services, which provide a web service over an ODI data store (i.e. a table, view or other data source registered in ODI)
• The ODIInvokeWebService tool that you can add to a package to request a response from a web service

11) Where does ODI sit with my existing OWB implementation(s)?
As mentioned previously, the ODI-EE licence includes both ODI and OWB as separate products, both tools will converge in time into "Oracle’s Unified Data Integration Product".

Oracle have released a statement of direction for both products, published January 2010:

http://www.oracle.com/technology/products/oracle-data-integrator/sod.pdf

OWB 11G R2 is the first step from Oracle to bring these two applications together, its now possible to use ODI Knowledge modules within your OWB 11G R2 environment as 'Code Templates', an Oracle white paper published February 2010 describes this in more detail:

http://www.oracle.com/technology/products/warehouse/pdf/owb-11gr2-code-template-mappings.pdf

12) Is ODI Used by Oracle in their products?
Yes there are many Oracle products that utilise ODI, but here are just a few:

• Oracle Application Integration Architecture (AIA)
• Oracle Agile products
• Oracle Hyperion Financial Management
• Oracle Hyperion Planning
• Oracle Fusion Governance, Risk & Compliance
• Oracle Business Activity Monitoring

Oracle BI Applications also uses ODI as its core ETL tool in place of Informatica , but only for one release of OBIA and when using a certain source system.

Future plans are to have ODI fully available through the OBIA offering.

Oracle Data Integrator (ODI) - Frequently Asked Questions (FAQ)

1) What is Oracle Data Integrator (ODI)?
Oracle acquired Sunopsis in 2006 and with it "Sunopsis Data Integrator".

Oracle Data Integrator (ODI) is an E-LT (Extract, Load and Transform) tool used for high-speed data movement between disparate systems.

The latest version, Oracle Data Integrator Enterprise Edition (ODI-EE) brings together "Oracle Data Integrator" and "Oracle Warehouse Builder" as separate components of a single product with a single licence.

2) What is E-LT?E-LT is an innovative approach to extracting, loading and Transforming data. Typically ETL application vendors have relied on costly heavyweight , mid-tier server to perform the transformations required when moving large volumes of data around the enterprise.

ODI delivers unique next-generation, Extract Load and Transform (E-LT) technology that improves performance and reduces data integration costs, even across heterogeneous systems by pushing the processing required down to the typically large and powerful database servers already in place within the enterprise.


3) What components make up Oracle Data Integrator?"Oracle Data Integrator" comprises of:

- Oracle Data Integrator + Topology Manager + Designer + Operator + Agent
- Oracle Data Quality for Data Integrator
- Oracle Data Profiling

4) What is Oracle Data Integration Suite?

Oracle data integration suite is a set of data management applications for building, deploying, and managing enterprise data integration solutions:

• Oracle Data Integrator Enterprise Edition
• Oracle Data Relationship Management
• Oracle Service Bus (limited use)
• Oracle BPEL (limited use)
• Oracle WebLogic Server (limited use)

Additional product options are:

• Oracle Goldengate
• Oracle Data Quality for Oracle Data Integrator (Trillium-based DQ)
• Oracle Data Profiling (Trillium based Data Profiling)
• ODSI (the former Aqualogic Data Services Platform)

5) What systems can ODI extract and load data into?
ODI brings true heterogeneous connectivity out-of-the-box, it can connect natively to Oracle, Sybase, MS SQL Server, MySQL, LDAP, DB2, PostgreSQL, Netezza.

It can also connect to any data source supporting JDBC, its possible even to use the Oracle BI Server as a data source using the jdbc driver that ships with BI Publisher

6) What are Knowledge Modules?
Knowledge Modules form the basis of 'plug-ins' that allow ODI to generate the relevant execution code , across technologies , to perform tasks in one of six areas, the six types of knowledge module consist of:

• Reverse-engineering knowledge modules are used for reading the table and other object metadata from source databases
• Journalizing knowledge modules record the new and changed data within either a single table or view or a consistent set of tables or views
• Loading knowledge modules are used for efficient extraction of data from source databases for loading into a staging area (database-specific bulk unload utilities can be used where available)
• Check knowledge modules are used for detecting errors in source data
• Integration knowledge modules are used for efficiently transforming data from staging area to the target tables, generating the optimized native SQL for the given database
• Service knowledge modules provide the ability to expose data as Web services

ODI ships with many knowledge modules out of the box, these are also extendable, they can modified within the ODI Designer module.

7) How do 'Contexts' work in ODI?
ODI offers a unique design approach through use of Contexts and Logical schemas. Imagine a development team, within the ODI Topology manager a senior developer can define the system architecture, connections, databases, data servers (tables etc) and so forth.

These objects are linked through contexts to 'logical' architecture objects that are then used by other developers to simply create interfaces using these logical objects, at run-time, on specification of a context within which to execute the interfaces, ODI will use the correct physical connections, databases + tables (source + target) linked the logical objects being used in those interfaces as defined within the environment Topology.


8) Does my ODI infrastructure require an Oracle database?
No, the ODI modular repositories (Master + and one of multiple Work repositories) can be installed on any database engine that supports ANSI ISO 89 syntax such as Oracle, Microsoft SQL Server, Sybase AS Enterprise, IBM DB2 UDB, IBM DB2/40.


9) Where can I get more information on ODI?
The OTN Data integration home page : http://www.oracle.com/us/products/middleware/data-integration/index.html

10) Does ODI support web services?
Yes, ODI is 'SOA' enabled and its web services can be used in 3 ways:

• The Oracle Data Integrator Public Web Service, that lets you execute a scenario (a published package) from a web service call
• Data Services, which provide a web service over an ODI data store (i.e. a table, view or other data source registered in ODI)
• The ODIInvokeWebService tool that you can add to a package to request a response from a web service

11) Where does ODI sit with my existing OWB implementation(s)?
As mentioned previously, the ODI-EE licence includes both ODI and OWB as separate products, both tools will converge in time into "Oracle’s Unified Data Integration Product".

Oracle have released a statement of direction for both products, published January 2010:

http://www.oracle.com/technology/products/oracle-data-integrator/sod.pdf

OWB 11G R2 is the first step from Oracle to bring these two applications together, its now possible to use ODI Knowledge modules within your OWB 11G R2 environment as 'Code Templates', an Oracle white paper published February 2010 describes this in more detail:

http://www.oracle.com/technology/products/warehouse/pdf/owb-11gr2-code-template-mappings.pdf

12) Is ODI Used by Oracle in their products?
Yes there are many Oracle products that utilise ODI, but here are just a few:

• Oracle Application Integration Architecture (AIA)
• Oracle Agile products
• Oracle Hyperion Financial Management
• Oracle Hyperion Planning
• Oracle Fusion Governance, Risk & Compliance
• Oracle Business Activity Monitoring

Oracle BI Applications also uses ODI as its core ETL tool in place of Informatica , but only for one release of OBIA and when using a certain source system.

Future plans are to have ODI fully available through the OBIA offering.

Oracle Data Integrator (ODI) - Frequently Asked Questions (FAQ)

1) What is Oracle Data Integrator (ODI)?
Oracle acquired Sunopsis in 2006 and with it "Sunopsis Data Integrator".

Oracle Data Integrator (ODI) is an E-LT (Extract, Load and Transform) tool used for high-speed data movement between disparate systems.

The latest version, Oracle Data Integrator Enterprise Edition (ODI-EE) brings together "Oracle Data Integrator" and "Oracle Warehouse Builder" as separate components of a single product with a single licence.

2) What is E-LT?E-LT is an innovative approach to extracting, loading and Transforming data. Typically ETL application vendors have relied on costly heavyweight , mid-tier server to perform the transformations required when moving large volumes of data around the enterprise.

ODI delivers unique next-generation, Extract Load and Transform (E-LT) technology that improves performance and reduces data integration costs, even across heterogeneous systems by pushing the processing required down to the typically large and powerful database servers already in place within the enterprise.

3) What components make up Oracle Data Integrator?"Oracle Data Integrator" comprises of:

- Oracle Data Integrator + Topology Manager + Designer + Operator + Agent
- Oracle Data Quality for Data Integrator
- Oracle Data Profiling

4) What is Oracle Data Integration Suite?

Oracle data integration suite is a set of data management applications for building, deploying, and managing enterprise data integration solutions:

• Oracle Data Integrator Enterprise Edition
• Oracle Data Relationship Management
• Oracle Service Bus (limited use)
• Oracle BPEL (limited use)
• Oracle WebLogic Server (limited use)

Additional product options are:

• Oracle Goldengate
• Oracle Data Quality for Oracle Data Integrator (Trillium-based DQ)
• Oracle Data Profiling (Trillium based Data Profiling)
• ODSI (the former Aqualogic Data Services Platform)

5) What systems can ODI extract and load data into?
ODI brings true heterogeneous connectivity out-of-the-box, it can connect natively to Oracle, Sybase, MS SQL Server, MySQL, LDAP, DB2, PostgreSQL, Netezza.

It can also connect to any data source supporting JDBC, its possible even to use the Oracle BI Server as a data source using the jdbc driver that ships with BI Publisher

6) What are Knowledge Modules?
Knowledge Modules form the basis of 'plug-ins' that allow ODI to generate the relevant execution code , across technologies , to perform tasks in one of six areas, the six types of knowledge module consist of:

• Reverse-engineering knowledge modules are used for reading the table and other object metadata from source databases
• Journalizing knowledge modules record the new and changed data within either a single table or view or a consistent set of tables or views
• Loading knowledge modules are used for efficient extraction of data from source databases for loading into a staging area (database-specific bulk unload utilities can be used where available)
• Check knowledge modules are used for detecting errors in source data
• Integration knowledge modules are used for efficiently transforming data from staging area to the target tables, generating the optimized native SQL for the given database
• Service knowledge modules provide the ability to expose data as Web services

ODI ships with many knowledge modules out of the box, these are also extendable, they can modified within the ODI Designer module.

7) How do 'Contexts' work in ODI?
ODI offers a unique design approach through use of Contexts and Logical schemas. Imagine a development team, within the ODI Topology manager a senior developer can define the system architecture, connections, databases, data servers (tables etc) and so forth.

These objects are linked through contexts to 'logical' architecture objects that are then used by other developers to simply create interfaces using these logical objects, at run-time, on specification of a context within which to execute the interfaces, ODI will use the correct physical connections, databases + tables (source + target) linked the logical objects being used in those interfaces as defined within the environment Topology.


8) Does my ODI infrastructure require an Oracle database?
No, the ODI modular repositories (Master + and one of multiple Work repositories) can be installed on any database engine that supports ANSI ISO 89 syntax such as Oracle, Microsoft SQL Server, Sybase AS Enterprise, IBM DB2 UDB, IBM DB2/40.


9) Where can I get more information on ODI?
The OTN Data integration home page : http://www.oracle.com/us/products/middleware/data-integration/index.html

10) Does ODI support web services?
Yes, ODI is 'SOA' enabled and its web services can be used in 3 ways:

• The Oracle Data Integrator Public Web Service, that lets you execute a scenario (a published package) from a web service call
• Data Services, which provide a web service over an ODI data store (i.e. a table, view or other data source registered in ODI)
• The ODIInvokeWebService tool that you can add to a package to request a response from a web service

11) Where does ODI sit with my existing OWB implementation(s)?
As mentioned previously, the ODI-EE licence includes both ODI and OWB as separate products, both tools will converge in time into "Oracle’s Unified Data Integration Product".

Oracle have released a statement of direction for both products, published January 2010:

http://www.oracle.com/technology/products/oracle-data-integrator/sod.pdf

OWB 11G R2 is the first step from Oracle to bring these two applications together, its now possible to use ODI Knowledge modules within your OWB 11G R2 environment as 'Code Templates', an Oracle white paper published February 2010 describes this in more detail:

http://www.oracle.com/technology/products/warehouse/pdf/owb-11gr2-code-template-mappings.pdf

12) Is ODI Used by Oracle in their products?
Yes there are many Oracle products that utilise ODI, but here are just a few:

• Oracle Application Integration Architecture (AIA)
• Oracle Agile products
• Oracle Hyperion Financial Management
• Oracle Hyperion Planning
• Oracle Fusion Governance, Risk & Compliance
• Oracle Business Activity Monitoring

Oracle BI Applications also uses ODI as its core ETL tool in place of Informatica , but only for one release of OBIA and when using a certain source system.

Future plans are to have ODI fully available through the OBIA offering.

Oracle Data Integrator (ODI) - Frequently Asked Questions (FAQ)

1) What is Oracle Data Integrator (ODI)?
Oracle acquired Sunopsis in 2006 and with it "Sunopsis Data Integrator".

Oracle Data Integrator (ODI) is an E-LT (Extract, Load and Transform) tool used for high-speed data movement between disparate systems.

The latest version, Oracle Data Integrator Enterprise Edition (ODI-EE) brings together "Oracle Data Integrator" and "Oracle Warehouse Builder" as separate components of a single product with a single licence.

2) What is E-LT?E-LT is an innovative approach to extracting, loading and Transforming data. Typically ETL application vendors have relied on costly heavyweight , mid-tier server to perform the transformations required when moving large volumes of data around the enterprise.

ODI delivers unique next-generation, Extract Load and Transform (E-LT) technology that improves performance and reduces data integration costs, even across heterogeneous systems by pushing the processing required down to the typically large and powerful database servers already in place within the enterprise.

3) What components make up Oracle Data Integrator?"Oracle Data Integrator" comprises of:

- Oracle Data Integrator + Topology Manager + Designer + Operator + Agent
- Oracle Data Quality for Data Integrator
- Oracle Data Profiling

4) What is Oracle Data Integration Suite?

Oracle data integration suite is a set of data management applications for building, deploying, and managing enterprise data integration solutions:

• Oracle Data Integrator Enterprise Edition
• Oracle Data Relationship Management
• Oracle Service Bus (limited use)
• Oracle BPEL (limited use)
• Oracle WebLogic Server (limited use)

Additional product options are:

• Oracle Goldengate
• Oracle Data Quality for Oracle Data Integrator (Trillium-based DQ)
• Oracle Data Profiling (Trillium based Data Profiling)
• ODSI (the former Aqualogic Data Services Platform)

5) What systems can ODI extract and load data into?
ODI brings true heterogeneous connectivity out-of-the-box, it can connect natively to Oracle, Sybase, MS SQL Server, MySQL, LDAP, DB2, PostgreSQL, Netezza.

It can also connect to any data source supporting JDBC, its possible even to use the Oracle BI Server as a data source using the jdbc driver that ships with BI Publisher

6) What are Knowledge Modules?
Knowledge Modules form the basis of 'plug-ins' that allow ODI to generate the relevant execution code , across technologies , to perform tasks in one of six areas, the six types of knowledge module consist of:

• Reverse-engineering knowledge modules are used for reading the table and other object metadata from source databases
• Journalizing knowledge modules record the new and changed data within either a single table or view or a consistent set of tables or views
• Loading knowledge modules are used for efficient extraction of data from source databases for loading into a staging area (database-specific bulk unload utilities can be used where available)
• Check knowledge modules are used for detecting errors in source data
• Integration knowledge modules are used for efficiently transforming data from staging area to the target tables, generating the optimized native SQL for the given database
• Service knowledge modules provide the ability to expose data as Web services

ODI ships with many knowledge modules out of the box, these are also extendable, they can modified within the ODI Designer module.

7) How do 'Contexts' work in ODI?
ODI offers a unique design approach through use of Contexts and Logical schemas. Imagine a development team, within the ODI Topology manager a senior developer can define the system architecture, connections, databases, data servers (tables etc) and so forth.

These objects are linked through contexts to 'logical' architecture objects that are then used by other developers to simply create interfaces using these logical objects, at run-time, on specification of a context within which to execute the interfaces, ODI will use the correct physical connections, databases + tables (source + target) linked the logical objects being used in those interfaces as defined within the environment Topology.


8) Does my ODI infrastructure require an Oracle database?
No, the ODI modular repositories (Master + and one of multiple Work repositories) can be installed on any database engine that supports ANSI ISO 89 syntax such as Oracle, Microsoft SQL Server, Sybase AS Enterprise, IBM DB2 UDB, IBM DB2/40.


9) Where can I get more information on ODI?
The OTN Data integration home page : http://www.oracle.com/us/products/middleware/data-integration/index.html

10) Does ODI support web services?
Yes, ODI is 'SOA' enabled and its web services can be used in 3 ways:

• The Oracle Data Integrator Public Web Service, that lets you execute a scenario (a published package) from a web service call
• Data Services, which provide a web service over an ODI data store (i.e. a table, view or other data source registered in ODI)
• The ODIInvokeWebService tool that you can add to a package to request a response from a web service

11) Where does ODI sit with my existing OWB implementation(s)?
As mentioned previously, the ODI-EE licence includes both ODI and OWB as separate products, both tools will converge in time into "Oracle’s Unified Data Integration Product".

Oracle have released a statement of direction for both products, published January 2010:

http://www.oracle.com/technology/products/oracle-data-integrator/sod.pdf

OWB 11G R2 is the first step from Oracle to bring these two applications together, its now possible to use ODI Knowledge modules within your OWB 11G R2 environment as 'Code Templates', an Oracle white paper published February 2010 describes this in more detail:

http://www.oracle.com/technology/products/warehouse/pdf/owb-11gr2-code-template-mappings.pdf

12) Is ODI Used by Oracle in their products?
Yes there are many Oracle products that utilise ODI, but here are just a few:

• Oracle Application Integration Architecture (AIA)
• Oracle Agile products
• Oracle Hyperion Financial Management
• Oracle Hyperion Planning
• Oracle Fusion Governance, Risk & Compliance
• Oracle Business Activity Monitoring

Oracle BI Applications also uses ODI as its core ETL tool in place of Informatica , but only for one release of OBIA and when using a certain source system.

Future plans are to have ODI fully available through the OBIA offering.