Protocol Documentation

Table of Contents

api/datamovement.proto

Top

DataMovementCancelRequest

The Data Movement Cancel Request permits users to initiate a cancellation of a data movement request that is in progress.

FieldTypeLabelDescription
workflow DataMovementWorkflow

The name and namespace of the initiating workflow

uid string

The unique identifier for the data movement resource

DataMovementCancelResponse

The Data Movement Cancel Response returns the status of the cancel request. The cancel process will begin upon success. This response does not indicate that that cancel process has completed, but rather that it has been initiated.

FieldTypeLabelDescription
status DataMovementCancelResponse.Status

Status of the Data Movement Cancellation Request

message string

Any extra information included with the status

DataMovementCommandStatus

Data movement operations contain a CommandStatus in the Status object. This information relays the current state of the data movement progress.

FieldTypeLabelDescription
command string

Command used to perform Data Movement

progress int32

Progress percentage (0-100%) of the data movement based on the output of `dcp`. `dcp` must be present in the command.

elapsedTime string

Duration of how long the operation has been running (e.g. 1m30s)

lastMessage string

The most recent line of output from the data movement command

lastMessageTime string

The time (local) of lastMessage

DataMovementCreateRequest

Create a Data Movement Request to perform data movement. NNF Data Movement will copy from source to the destination path.

FieldTypeLabelDescription
workflow DataMovementWorkflow

The name and namespace of the initiating workflow

source string

Source file or directory

destination string

Destination file or directory

dryrun bool

If True, the data movement command runs `/bin/true` rather than perform actual data movement

dcpOptions string

Extra options to pass to `dcp` if present in the Data Movement command.

logStdout bool

If true, enable server-side logging of stdout when the command is successful. Failure output is always logged.

storeStdout bool

If true, store stdout in DataMovementStatusResponse.Message when the command is successful. Failure output is always contained in the message.

slots int32

The number of slots specified in the MPI hostfile. A value of 0 disables the use of slots in the hostfile. -1 will defer to the server side configuration.

maxSlots int32

The number of max_slots specified in the MPI hostfile. A value of 0 disables the use of max_slots in the hostfile. -1 will defer to the server side configuration.

profile string

The name of the data movement configuration profile to use. The above parameters (e.g. slots, logStdout) will override the settings defined by the profile. This profile must exist on the server otherwise the data movement operation will be invalid. Empty will default to the default profile.

mpirunOptions string

Extra options to pass to `mpirun` if present in the Data Movement command.

DataMovementCreateResponse

The Data Movement Create Response to indicate the status of of the Data Movement Request.

FieldTypeLabelDescription
uid string

The unique identifier for the created data movement resource if `Status` is Success

status DataMovementCreateResponse.Status

Status of the DataMovementCreateRequest

message string

Any extra information supplied with the status

DataMovementDeleteRequest

The Data Movement Delete Request permits users to delete a completed data movement request (any request with a status of COMPLETED).

FieldTypeLabelDescription
workflow DataMovementWorkflow

The name and namespace of the initiating workflow

uid string

The unique identifier for the data movement resource

DataMovementDeleteResponse

The Data Movement Delete Response returns the status of the delete operation.

FieldTypeLabelDescription
status DataMovementDeleteResponse.Status

Status of the Data Movement Delete Request

message string

Any extra information supplied with the status

DataMovementListRequest

The Data Movement List Request allows a user to get a list of the current data movement requests for a given DWS Workflow.

FieldTypeLabelDescription
workflow DataMovementWorkflow

The name and namespace of the initiating workflow

DataMovementListResponse

The Data Movement List Response returns a list of the matching data movement requests' UIDs

FieldTypeLabelDescription
uids string repeated

List of data movement requests associated with the given workflow and namespace

DataMovementStatusRequest

The Data Movement Status Request message permits users to query the status of a Data Movement Request.

FieldTypeLabelDescription
workflow DataMovementWorkflow

The name and namespace of the initiating workflow

uid string

UID of the Data Movement Request

maxWaitTime int64

The maximum time in seconds to wait for completion of the data movement resource. Negative values imply an indefinite wait

DataMovementStatusResponse

The Data Movement Status Response message defines the current status of a data movement request.

FieldTypeLabelDescription
state DataMovementStatusResponse.State

Current state of the Data Movement Request

status DataMovementStatusResponse.Status

Current status of the Data Movement Request

message string

Any extra information supplied with the state/status

commandStatus DataMovementCommandStatus

Current state/progress of the Data Movement command

startTime string

The start time (local) of the data movement operation

endTime string

The end time (local) of the data movement operation

DataMovementVersionResponse

The data movement version response returns information on the running data movement daemon.

FieldTypeLabelDescription
version string

Current version of the API.

apiVersions string repeated

List of supported API versions.

DataMovementWorkflow

Data Movement workflow message contains identifying information for the workflow initiating the data movement operation. This message must be nested in all Request messages.

FieldTypeLabelDescription
name string

Name of the DWS workflow that is associated with this data movement request

namespace string

Namespace of the DWS workflow that is associated with this data movement request

DataMovementCancelResponse.Status

NameNumberDescription
INVALID 0

The cancel request was found to be invalid

NOT_FOUND 1

The request with the supplied UID was not found

SUCCESS 2

The data movement request was cancelled successfully

FAILED 3

The data movement request cannot be canceled

DataMovementCreateResponse.Status

NameNumberDescription
SUCCESS 0

The data movement resource created successfully

FAILED 1

The data movement resource failed to create. See `message` field for more information

INVALID 2

Invalid request

DataMovementDeleteResponse.Status

NameNumberDescription
INVALID 0

The delete request was found to be invalid

NOT_FOUND 1

The request with the supplied UID was not found

SUCCESS 2

The data movement request was deleted successfully

ACTIVE 3

The data movement request is currently active and cannot be deleted

UNKNOWN 4

Unknown status

DataMovementStatusResponse.State

NameNumberDescription
PENDING 0

The request is created but has a pending status

STARTING 1

The request was created and is in the act of starting

RUNNING 2

The data movement request is actively running

COMPLETED 3

The data movement request has completed

CANCELLING 4

The data movement request has started the cancellation process

UNKNOWN_STATE 5

Unknown state

DataMovementStatusResponse.Status

NameNumberDescription
INVALID 0

The request was found to be invalid. See `message` for details

NOT_FOUND 1

The request with the supplied UID was not found

SUCCESS 2

The request completed with success

FAILED 3

The request failed. See `message` for details

CANCELLED 4

The request was cancelled

UNKNOWN_STATUS 5

Unknown status

DataMover

DataMover service definition describes the API for perform data movement for NNF storage.

Method NameRequest TypeResponse TypeDescription
Version .google.protobuf.Empty DataMovementVersionResponse

Version sends a request to the data movement daemon and returns a response containing details on the current build version and supported API versions.

Create DataMovementCreateRequest DataMovementCreateResponse

Create sends a new data movement request identified by source and destination fields. It returns a response containing a unique identifier which can be used to query the status of the request.

Status DataMovementStatusRequest DataMovementStatusResponse

Status requests the status of a previously submitted data movement request. It accepts a unique identifier that identifies the request and returns a status message.

Delete DataMovementDeleteRequest DataMovementDeleteResponse

Delete will attempt to delete a completed data movement request. It accepts a unique identifier that identifies the request and returns the status of the delete operation.

List DataMovementListRequest DataMovementListResponse

List returns all current data movement requests for a given namespace and workflow.

Cancel DataMovementCancelRequest DataMovementCancelResponse

Cancel will attempt to stop a data movement request. It accepts a unique identifier that identifies the request and returns the status of the cancel operation.

Scalar Value Types

.proto TypeNotesC++JavaPythonGoC#PHPRuby
double double double float float64 double float Float
float float float float float32 float float Float
int32 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int int32 int integer Bignum or Fixnum (as required)
int64 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long int64 long integer/string Bignum
uint32 Uses variable-length encoding. uint32 int int/long uint32 uint integer Bignum or Fixnum (as required)
uint64 Uses variable-length encoding. uint64 long int/long uint64 ulong integer/string Bignum or Fixnum (as required)
sint32 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int int32 int integer Bignum or Fixnum (as required)
sint64 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long int64 long integer/string Bignum
fixed32 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int uint32 uint integer Bignum or Fixnum (as required)
fixed64 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long uint64 ulong integer/string Bignum
sfixed32 Always four bytes. int32 int int int32 int integer Bignum or Fixnum (as required)
sfixed64 Always eight bytes. int64 long int/long int64 long integer/string Bignum
bool bool boolean boolean bool bool boolean TrueClass/FalseClass
string A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode string string string String (UTF-8)
bytes May contain any arbitrary sequence of bytes. string ByteString str []byte ByteString string String (ASCII-8BIT)