EDP job engine objects provide methods for creating, monitoring, and terminating jobs on Sahara clusters. Provisioning plugins that support EDP must return an EDP job engine object from the get_edp_engine(cluster, job_type) method described in Plugin SPI.
Sahara provides subclasses of the base job engine interface that support EDP on clusters running Oozie or on Spark standalone clusters. These are described below.
Some of the methods below test job type. Sahara supports the following string values for job types:
Note, constants for job types are defined in sahara.utils.edp
Several of the methods below return a job status value. A job status value is a dictionary of the form:
{‘status’: job_status_value}
where job_status_value is one of the following string values:
Note, constants for job status are defined in sahara.utils.edp
The sahara.service.edp.base_engine.JobEngine class is an abstract class with the following interface:
Stops the running job whose id is stored in the job_execution object.
Returns: None if the operation was unsuccessful or an updated job status value
Returns the current status of the job whose id is stored in the job_execution object.
Returns: a job status value
Starts the job described by the job_execution object
Returns: a tuple of the form (job_id, job_status_value, job_extra_info)
Checks whether or not the job can run on the cluster with the specified data. Data contains values passed to the /jobs/<job_id>/execute REST API method during job launch. If the job cannot run for any reason, including job configuration, cluster configuration, or invalid data, this method should raise an exception.
Returns: None
Returns hints used by the Sahara UI to prompt users for values when configuring and launching a job. Note that no hints are required.
See Elastic Data Processing (EDP) for more information on how configuration values, parameters, and arguments are used by different job types.
Returns: a dictionary of the following form, containing hints for configs, parameters, and arguments for the job type:
{‘job_config’: {‘configs’: [], ‘params’: {}, ‘args’: []}}
This method returns the job types that the engine supports. Not all engines will support all job types.
Returns: a list of job types supported by the engine
The sahara.service.edp.oozie.engine.OozieJobEngine class is derived from JobEngine. It provides implementations for all of the methods in the base interface but adds a few more abstract methods.
Note, the validate_job_execution(cluster, job, data) method does basic checks on the job configuration but probably should be overloaded to include additional checks on the cluster configuration. For example, the job engines for plugins that support Oozie add checks to make sure that the Oozie service is up and running.
Oozie uses HDFS to distribute job files. This method gives the name of the account that is used on the data nodes to access HDFS (such as ‘hadoop’ or ‘hdfs’). The Oozie job engine expects that HDFS contains a directory for this user under /user/
Returns: a string giving the username for the account used to access HDFS on the cluster.
The remote object remote references a node in the cluster. This method creates the HDFS directory dir_name under the user specified by get_hdfs_user() in the HDFS accessible from the specified node. For example, if the HDFS user is ‘hadoop’ and the dir_name is ‘test’ this method would create ‘/user/hadoop/test’.
The reason that this method is broken out in the interface as an abstract method is that different versions of Hadoop treat path creation differently.
Returns: None
Returns the full URI for the Oozie server, for example http://my_oozie_host:11000/oozie. This URI is used by an Oozie client to send commands and queries to the Oozie server.
Returns: a string giving the Oozie server URI.
Returns the node instance for the host in the cluster running the Oozie server
Returns: a node instance
Returns the full URI for the Hadoop NameNode, for example http://master_node:8020.
Returns: a string giving the NameNode URI.
Returns the full URI for the Hadoop JobTracker for Hadoop version 1 or the Hadoop ResourceManager for Hadoop version 2.
Returns: a string giving the JobTracker or ResourceManager URI
The sahara.service.edp.spark.engine.SparkJobEngine class provides a full EDP implementation for Spark standalone clusters.
Note, the validate_job_execution(cluster, job, data) method does basic checks on the job configuration but probably should be overloaded to include additional checks on the cluster configuration. For example, the job engine returned by the Spark plugin checks that the Spark version is >= 1.0.0 to ensure that spark-submit is available.
Returns driver class path.
Returns: a string of the following format ‘ –driver-class-path class_path_value‘