Tuesday, June 18, 2013

Apache Oozie - Part 11: Java API for interfacing with oozie workflows


What's covered in the blog?

1. Documentation on the Oozie java API
2. Code of a sample java program that calls a oozie workflow with a java action to process some syslog generated log files.  Instructions on loading sample data, workflow files and running the workflow are provided, along with some notes based on my learnings.

Version:
Oozie 3.3.0;

Related blogs:

Your thoughts/updates:
If you want to share your thoughts/updates, email me at airawat.blog@gmail.com.

1.0. About the Oozie java API

Oozie provides a Java Client API that simplifies integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services API.
The following code snippet shows how to submit an Oozie job using the Java Client API.

import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
.
import java.util.Properties;
.
    ...
.
    // get a OozieClient for local Oozie
    OozieClient wc = new OozieClient("http://bar:11000/oozie");
.
    // create a workflow job configuration and set the workflow application path
    Properties conf = wc.createConfiguration();
    conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
.
    // setting workflow parameters
    conf.setProperty("jobTracker", "foo:8021");
    conf.setProperty("inputDir", "/usr/tucu/inputdir");
    conf.setProperty("outputDir", "/usr/tucu/outputdir");
    ...
.
    // submit and start the workflow job
    String jobId = wc.run(conf);
    System.out.println("Workflow job submitted");
.
    // wait until the workflow job finishes printing the status every 10 seconds
    while (wc.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) {
        System.out.println("Workflow job running ...");
        Thread.sleep(10 * 1000);
    }
.
    // print the final status o the workflow job
    System.out.println("Workflow job completed ...");
    System.out.println(wf.getJobInfo(jobId));
    ...

Source of the documentation, above:
http://archive.cloudera.com/cdh/3/oozie/DG_Examples.html#Java_API_Example


2.0. Exercise

The java program below calls the workflow built in my blog 6 - include java code, workflow related files and sample data.

2.0.1. Sample data and sample workflow


2.0.2. Sample Java program to call workflow

Note: Ensure you replace configuration highlighted in yellow ochre with your cluster specific configuration.

import java.util.Properties;

import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;

public class myOozieWorkflowJavaAPICall {

public static void main(String[] args) {
    OozieClient wc = new OozieClient("http://cdh-dev01:11000/oozie");

    Properties conf = wc.createConfiguration();

    conf.setProperty(OozieClient.APP_PATH, "hdfs://cdh-nn01.hadoop.com:8020/user/airawat/oozieProject/javaApplication/workflow.xml");
    conf.setProperty("jobTracker", "cdh-jt01:8021");
    conf.setProperty("nameNode", "hdfs://cdh-nn01.hadoop.com:8020");
    conf.setProperty("queueName", "default");
    conf.setProperty("airawatOozieRoot", "hdfs://cdh-nn01.hadoop.com:8020/user/airawat/oozieProject/javaApplication");
    conf.setProperty("oozie.libpath", "hdfs://cdh-nn01.hadoop.com:8020/user/oozie/share/lib");
    conf.setProperty("oozie.use.system.libpath", "true");
    conf.setProperty("oozie.wf.rerun.failnodes", "true");

    try {
        String jobId = wc.run(conf);
        System.out.println("Workflow job, " + jobId + " submitted");

        while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
            System.out.println("Workflow job running ...");
            Thread.sleep(10 * 1000);
        }
        System.out.println("Workflow job completed ...");
System.out.println(wc.getJobInfo(jobId));
    } catch (Exception r) {
        System.out.println("Errors " + r.getLocalizedMessage());
    }
}
}

2.0.3. Program output

Workflow job, 0000081-130613112811513-oozie-oozi-W submitted
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job completed ...
Workflow id[0000081-130613112811513-oozie-oozi-W] status[SUCCEEDED]

2.0.4. Oozie web console

http://YourOozieServer:TypicallyPort11000/oozie


















27 comments:

  1. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    Fita Chennai reviews
    Hadoop Training in Chennai
    Big Data Training in Chennai

    ReplyDelete
  2. Thanks for your informative article on ios mobile application development. Your article helped me to explore the future of mobile apps developers. Having sound knowledge on mobile application development will help you to float in mobile application development. iOS Training in Chennai | iOS Training Institutes in Chennai

    ReplyDelete
  3. Great article. Glad to find your blog. Thanks for sharing.
    perl training in chennai

    ReplyDelete
  4. Your blog is really nice and informative. Thanks for sharing this post. Keep posting.
    ssas training in chennai

    ReplyDelete
  5. Hi,
    Thanks for the info. I'm trying the Oozie Java Api in Kerberos Environment. Do you have any example which has kerberos authentication for Oozie Java Api. Kindly share if you have any.

    ReplyDelete
    Replies
    1. use AuthOozieClient method to invoke client,

      AuthOozieClient wc = new AuthOozieClient("http://:11000/oozie","KERBEROS");

      Thanks,
      Sathish

      Delete
  6. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  7. Is there anyway that you could post the libraries that u use in this program please? Thank you

    ReplyDelete
  8. Your article on iOS was excellent with the advanced features and the scope of iOS developer in future what you said is absolutely correct. I would like to share with my friends so that they will also get the chance to read the blog.

    Regards:

    iOS Training in Chennai |
    iOS Training Institutes in Chennai

    ReplyDelete
  9. Hi Admin,
    I went through your blog and it’s totally awesome. Keep on updating your site with such informative post. If possible please include rss feed for your blog. Web design training in Chennai | Web designing course in Chennai | Best web designing course in Chennai

    ReplyDelete
  10. Great and impressive article!! Got to learn and know more about web development. To know more refer create website for much more unique ideas.PHP Training in Chennai | Best PHP training in Chennai

    ReplyDelete
  11. thakyou it vry nice blog for beginners
    https://www.emexotechnologies.com/courses/big-data-analytics-training/big-data-hadoop-training/

    ReplyDelete
  12. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    https://www.emexotechnologies.com/online-courses/big-data-hadoop-training-in-electronic-city/

    ReplyDelete
  13. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    Big Data Hadoop training in electronic city

    ReplyDelete
  14. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Data Science training in Chennai | Data science training in bangalore

    Data science training in pune | Data science online training

    Data Science Interview questions and answers

    ReplyDelete
  15. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.
    Best Devops Training in pune

    ReplyDelete
  16. I prefer to study this kind of material. Nicely written information in this post, the quality of content is fine and the conclusion is lovely. Things are very open and intensely clear explanation of issues
    python training in chennai | python training in chennai | python training in bangalore

    ReplyDelete
  17. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Java training in Chennai | Java training in Annanagar | Java training in Chennai

    Java training in Chennai | Java training in Bangalore | Java training in Electronic city

    ReplyDelete
  18. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.
    Devops Training courses
    Devops Training in Bangalore
    Best Devops Training in pune
    Devops interview questions and answers

    ReplyDelete