Tuesday, June 18, 2013

Apache Oozie - Part 11: Java API for interfacing with oozie workflows


What's covered in the blog?

1. Documentation on the Oozie java API
2. Code of a sample java program that calls a oozie workflow with a java action to process some syslog generated log files.  Instructions on loading sample data, workflow files and running the workflow are provided, along with some notes based on my learnings.

Version:
Oozie 3.3.0;

Related blogs:

Your thoughts/updates:
If you want to share your thoughts/updates, email me at airawat.blog@gmail.com.

1.0. About the Oozie java API

Oozie provides a Java Client API that simplifies integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services API.
The following code snippet shows how to submit an Oozie job using the Java Client API.

import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
.
import java.util.Properties;
.
    ...
.
    // get a OozieClient for local Oozie
    OozieClient wc = new OozieClient("http://bar:11000/oozie");
.
    // create a workflow job configuration and set the workflow application path
    Properties conf = wc.createConfiguration();
    conf.setProperty(OozieClient.APP_PATH, "hdfs://foo:8020/usr/tucu/my-wf-app");
.
    // setting workflow parameters
    conf.setProperty("jobTracker", "foo:8021");
    conf.setProperty("inputDir", "/usr/tucu/inputdir");
    conf.setProperty("outputDir", "/usr/tucu/outputdir");
    ...
.
    // submit and start the workflow job
    String jobId = wc.run(conf);
    System.out.println("Workflow job submitted");
.
    // wait until the workflow job finishes printing the status every 10 seconds
    while (wc.getJobInfo(jobId).getStatus() == Workflow.Status.RUNNING) {
        System.out.println("Workflow job running ...");
        Thread.sleep(10 * 1000);
    }
.
    // print the final status o the workflow job
    System.out.println("Workflow job completed ...");
    System.out.println(wf.getJobInfo(jobId));
    ...

Source of the documentation, above:
http://archive.cloudera.com/cdh/3/oozie/DG_Examples.html#Java_API_Example


2.0. Exercise

The java program below calls the workflow built in my blog 6 - include java code, workflow related files and sample data.

2.0.1. Sample data and sample workflow


2.0.2. Sample Java program to call workflow

Note: Ensure you replace configuration highlighted in yellow ochre with your cluster specific configuration.

import java.util.Properties;

import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;

public class myOozieWorkflowJavaAPICall {

public static void main(String[] args) {
    OozieClient wc = new OozieClient("http://cdh-dev01:11000/oozie");

    Properties conf = wc.createConfiguration();

    conf.setProperty(OozieClient.APP_PATH, "hdfs://cdh-nn01.hadoop.com:8020/user/airawat/oozieProject/javaApplication/workflow.xml");
    conf.setProperty("jobTracker", "cdh-jt01:8021");
    conf.setProperty("nameNode", "hdfs://cdh-nn01.hadoop.com:8020");
    conf.setProperty("queueName", "default");
    conf.setProperty("airawatOozieRoot", "hdfs://cdh-nn01.hadoop.com:8020/user/airawat/oozieProject/javaApplication");
    conf.setProperty("oozie.libpath", "hdfs://cdh-nn01.hadoop.com:8020/user/oozie/share/lib");
    conf.setProperty("oozie.use.system.libpath", "true");
    conf.setProperty("oozie.wf.rerun.failnodes", "true");

    try {
        String jobId = wc.run(conf);
        System.out.println("Workflow job, " + jobId + " submitted");

        while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
            System.out.println("Workflow job running ...");
            Thread.sleep(10 * 1000);
        }
        System.out.println("Workflow job completed ...");
System.out.println(wc.getJobInfo(jobId));
    } catch (Exception r) {
        System.out.println("Errors " + r.getLocalizedMessage());
    }
}
}

2.0.3. Program output

Workflow job, 0000081-130613112811513-oozie-oozi-W submitted
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job running ...
Workflow job completed ...
Workflow id[0000081-130613112811513-oozie-oozi-W] status[SUCCEEDED]

2.0.4. Oozie web console

http://YourOozieServer:TypicallyPort11000/oozie


















27 comments:

  1. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    Fita Chennai reviews
    Hadoop Training in Chennai
    Big Data Training in Chennai

    ReplyDelete
  2. Great article. Glad to find your blog. Thanks for sharing.
    perl training in chennai

    ReplyDelete
  3. Your blog is really nice and informative. Thanks for sharing this post. Keep posting.
    ssas training in chennai

    ReplyDelete
  4. Hi,
    Thanks for the info. I'm trying the Oozie Java Api in Kerberos Environment. Do you have any example which has kerberos authentication for Oozie Java Api. Kindly share if you have any.

    ReplyDelete
    Replies
    1. use AuthOozieClient method to invoke client,

      AuthOozieClient wc = new AuthOozieClient("http://:11000/oozie","KERBEROS");

      Thanks,
      Sathish

      Delete
  5. Is there anyway that you could post the libraries that u use in this program please? Thank you

    ReplyDelete
  6. Your article on iOS was excellent with the advanced features and the scope of iOS developer in future what you said is absolutely correct. I would like to share with my friends so that they will also get the chance to read the blog.

    Regards:

    iOS Training in Chennai |
    iOS Training Institutes in Chennai

    ReplyDelete
  7. thakyou it vry nice blog for beginners
    https://www.emexotechnologies.com/courses/big-data-analytics-training/big-data-hadoop-training/

    ReplyDelete
  8. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    https://www.emexotechnologies.com/online-courses/big-data-hadoop-training-in-electronic-city/

    ReplyDelete
  9. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    Big Data Hadoop training in electronic city

    ReplyDelete
  10. I prefer to study this kind of material. Nicely written information in this post, the quality of content is fine and the conclusion is lovely. Things are very open and intensely clear explanation of issues
    python training in chennai | python training in chennai | python training in bangalore

    ReplyDelete
  11. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Java training in Chennai | Java training in Annanagar | Java training in Chennai

    Java training in Chennai | Java training in Bangalore | Java training in Electronic city

    ReplyDelete
  12. Great article, valuable and excellent article, lots of great information, thanks for sharing with peoples.


    Data Science Bangalore

    ReplyDelete
  13. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    data analytics certification courses in Bangalore
    ExcelR Data science courses in Bangalore

    ReplyDelete
  14. Thank you so much for letting me express my feeling about your post.
    You write every blog post so well. Keep the hard work going and good luck.
    Hope to see such beneficial post ahead to.
    One data science
    One Machine Learning

    ReplyDelete
  15. You might comment on the order system of the blog. You should chat it's splendid. Your blog audit would swell up your visitors. I was very pleased to find this site.I wanted to thank you for this great read!!
    data analytics course malaysia

    ReplyDelete
  16. Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.
    Best PHP Training Institute in Chennai|PHP Course in chennai

    Best .Net Training Institute in Chennai
    Big Data Hadoop Training in Chennai
    Linux Training in Chennai
    Cloud Computing Training in Chennai

    ReplyDelete
  17. Thank you very much for providing important information. All your information is very valuable to me.
    Village Talkies a top-quality professional corporate video production company in Bangalore and also best explainer video company in Bangalore & animation video makers in Bangalore, Chennai, India & Maryland, Baltimore, USA provides Corporate & Brand films, Promotional, Marketing videos & Training videos, Product demo videos, Employee videos, Product video explainers, eLearning videos, 2d Animation, 3d Animation, Motion Graphics, Whiteboard Explainer videos Client Testimonial Videos, Video Presentation and more for all start-ups, industries, and corporate companies. From scripting to corporate video production services, explainer & 3d, 2d animation video production , our solutions are customized to your budget, timeline, and to meet the company goals and objectives.
    As a best video production company in Bangalore, we produce quality and creative videos to our clients.

    ReplyDelete
  18. Did you want to set your career towards Oracle? Then Infycle is with you to make this into reality. Infycle Technologies gives the combined and best Oracle course in Chennai, which offers various stages of Oracle such as Oracle PL/SQL, Oracle DBA, etc., along with 100% hands-on training guided by professional tutors in the field. Along with that, the mock interviews will be given to the candidates to face the interviews with complete confidence. Apart from all, the candidates will be placed in the top MNC's with an excellent salary package. To get it all, call 7502633633 and make this happen for your happy life.Best Oracle Training in Chennai | Infycle Technologies

    ReplyDelete
  19. I cannot thank you enough for the blog.Thanks Again. Keep writing.
    java course
    learn java online

    ReplyDelete