Skip to content

Using REST APis with Cloudera

by on May 18, 2015

We added a new function to the oozie and it forces us to do some extra post installation tasks:

  • Copy the jar to oozie read location
  • Update a few configuration parameters

Posting a question in Cloudera forum returned with a quick answer that helped me write the script below.

Note – I could have the script much shorter, if I created the required json file hard-coded, and just loaded it. But this was mre fun, and it may be useful if there are some changes in the oozie configuration in the next versions.

Once i got this work, my next target is to create a new full cluster only with REST apis.

Update – need to modify the script – have to add view=full, otherwise i get partial configuration and the below does not work.

#!/bin/bash
#
# This script modifies configuration in oozie for the new functionality
#
# It extract first the current configuration for oozie in json format
# Then it checks 3 parameters: xxxHadoopCounter, com.xxx.oozie.actions.WaitAction and wait-action-0.5.xsd
# If the parameter is missing it is added to the json file
#
# Then it loads the json back into the CDH
#
#
# Usage
#
function Usage {
echo "Usage: $0 "
exit 2
}
export host=$1
[[ -z "$host" ]] && Usage
#
# get_version
#
function get_version {
curl -u admin:admin "http://$host:7180/api/version" 2> /dev/null
}

export apiver=`get_version`
#
# get_cluster_name
#
function get_cluster_name {
curl -u admin:admin "http://$host:7180/api/$apiver/clusters" 2> /dev/null | awk '/name/{print $NF}'|sed 's/,//' | sed 's/"//g'
}

cname=`get_cluster_name`

tmp_json=/tmp/oozie_`date +%Y%m%d_%H%M%S`.json

#
# Extract the json
#
curl -u admin:admin http://$host:7180/api/${apiver}/clusters/${cname}/services/oozie/roleConfigGroups/oozie-OOZIE_SERVER-BASE/config > $tmp_json 2> /dev/null
cp    $tmp_json ${tmp_json}.ORIG

#
# 1. add the oozie_config_safety_valve
#
grep -l -s xxxHadoopCounter=com.xxx.oozie.CountersElFunctions#xxxHadoopCounter $tmp_json > /dev/null 2> /dev/null
if [ $? -ne 0 ]
then
awk '
/items/{
print $0;
print " \"name\" : \"oozie_config_safety_valve\",\n\"value\" : \"\\noozie.action.sharelib.for.java\\njava\\n\\n\\noozie.service.ELService.ext.functions.workflow\\nxxxHadoopCounter=com.xxx.oozie.CountersElFunctions#xxxHadoopCounter\\n\\n\"\n}, { " ; next ;
}
{print $0}
' $tmp_json > ${tmp_json}.new
mv ${tmp_json}.new $tmp_json
fi

#
# 2. add the oozie_executor_extension_classes
#
grep -l -s com.xxx.oozie.actions.WaitAction $tmp_json > /dev/null 2> /dev/null
if [ $? -ne 0 ]
then
awk '
/oozie_executor_extension_classes/{addvalue=",com.xxx.oozie.actions.WaitAction";}
/value" :/&&addvalue{sub(/\"$/,"");print $0 addvalue "\"" ;addvalue="";next}
{print $0}
' $tmp_json > ${tmp_json}.new
mv ${tmp_json}.new $tmp_json
fi

#
# 3. add the oozie_workflow_extension_schemas
#
grep -l -s wait-action-0.5.xsd $tmp_json > /dev/null 2> /dev/null
if [ $? -ne 0 ]
then
awk '
/oozie_workflow_extension_schemas/{addvalue=",wait-action-0.5.xsd";}
/value" :/&&addvalue{sub(/\"$/,"");print $0 addvalue "\"" ;addvalue="";next}
{print $0}
' $tmp_json > ${tmp_json}.new
mv ${tmp_json}.new $tmp_json
fi
curl -u admin:admin \
-H "Content-Type: application/json" \
-X PUT \
http://$host:7180/api/${ver}/clusters/${cname}/services/oozie/roleConfigGroups/oozie-OOZIE_SERVER-BASE/config \
-d "`cat ${tmp_json}`" > /dev/null 2> /dev/null
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: