curie.iogen package

Submodules

curie.iogen.fio_config module

class curie.iogen.fio_config.FioConfiguration(*args, **kwargs)

Bases: collections.OrderedDict

Manage an FIO configuration file.

DEFAULT_IODEPTH_PER_NODE = 4
convert_prefill(**kwargs)

Converts an existing configuration to a prefill configuration file.

Parameters:**kwargs – Keyword arguments can be used to override global defaults.
Returns:(FioConfiguration object) The converted config.
Return type:new_config
static dump(fio_config, fh)

Saves the configuration to the file handle.

Parameters:
  • fio_config – (curie.iogen.FioConfiguration) FIO Configuration.
  • fh – (filehandle) The file handle to save the config to.
static dumps(fio_config)

Provides the string representation of the configuration.

Parameters:fio_config – (curie.iogen.FioConfiguration) FIO Configuration.

Returns: (str) configuration

format(*args, **kwargs)

Fills in a configuration template with the provided argument data.

This function is basically a pass-through to Python’s string format function.

Returns: FioConfiguration object

static load(fh)

Loads a configuration file.

Parameters:fh – (file handle) reads the configuration from a file handle.

Returns: FioConfiguration object

static loads(string)

Loads a configuration from a string.

Parameters:string – (str) The configuration string to load.

Returns: FioConfiguration object

prefill_defaults = '[global]\nioengine=libaio\ndirect=1\nbs=1m\niodepth=4\nrw=write'
save(file_location)

Saves the configuration to the location specified.

Parameters:file_location – (str) The filepath to save the file to.
section_re = <_sre.SRE_Pattern object>
set(section_name, parameter, value=None)

Sets a parameter in a configuration.

Parameters:
  • section_name – (str) The name of the job/section to place the parameter.
  • parameter – (str) The name of the parameter.
  • value – (str) The value to set. May be None if the parameter does not require a value.

curie.iogen.fio_result module

class curie.iogen.fio_result.FioWorkloadResult(results_file)

Bases: object

An object to represent the results acquired from an FIO Workload using minimal (terse) reporting.

clear_cache()
get_active_series()

Provides a series that shows when the IO is active.

Returns: (pandas.Series) 1 where IO is active, 0 otherwise.

get_bytes_per_second_series()

Provides a bandwidth (bytes per second) series.

Returns: (pandas.Series) Bytes per second

get_errors_series()

Provides a total IOPS series.

Returns: (pandas.Series) Total errors.

get_io_field_series(job, field, subfield=None)

Acquires a field of data from all samples for a specific job and IO field.

Note: This relies upon the availability of timestamp_ms in the FIO JSON
output which is available in FIO releases > 2.10.
Parameters:
  • job – (str) FIO job index (typically 0 if group_reporting is used with only one group)
  • field – (str) “read”, “write”, “total_err”, “trim”, etc.
  • subfield – (str) Sub-field name to extract: e.g. “total_ios”

Returns: (pandas.Series) Series indexed by timestamp of the requested field.

get_iops_series()

Provides an IOPS series.

Returns: (pandas.Series) IOPS

get_results_data()

Provides the results data.

Parses raw file if necessary.

Returns: (list) A list of parsed JSON blocks (dicts).

get_series(result_type)
static series_to_result_pb(series)

Converts a Pandas series of data indexed by timestamps to a results protobuf.

Parameters:series – (pandas.Series) Series to convert.

Returns: (curie_test_pb2.CurieTestResult) test_result

curie.iogen.fio_workload module

class curie.iogen.fio_workload.FioWorkload(name, scenario, configuration, reporting_interval=5)

Bases: curie.iogen.iogen.IOGen

Implements a workload object for FIO.

Notes

  1. Terse output is used.
  2. Group reporting is currently required.
FIO_DEFAULT_GLOBAL_CONFIGURATION_PARAMS = {'continue_on_error': 'all', 'disk_util': 0, 'group_reporting': None}
FIO_DEFAULT_REPORTING_INTERVAL = 5
class curie.iogen.fio_workload.ResultResponse(success, data)

Bases: tuple

data

Alias for field number 1

success

Alias for field number 0

curie.iogen.iogen module

class curie.iogen.iogen.IOGen(name, scenario, configuration, short_name)

Bases: object

IOGen: Provides a basis for workload generator objects

An IOGen object is responsible for managing a workload generator:

  1. Configuration: setting and storing
  2. Results fetching
  3. Running

Note: Actual workload generators must implement certain functions to work properly, including:

  1. _cmd()
  2. _fetch_remote_results()
  3. _set_global_defaults()
fetch_remote_results(vms)

Acquires the results from the remote VM from the workload generator.

Parameters:vms – (list of curie.vm.Vm) VMs from which results are fetched.

Returns: List of tuples (success, results).

get_cmd_id()

Get the command ID for this workload generator.

Returns: (str) command id

get_local_results_path(vm)

Get the local results path for this workload generator and VM.

Parameters:vm – (curie.vm.Vm) VM associated with this results path.

Returns: (str) command id

get_workload_end_time()

Returns: (int) End time of the workload generator

get_workload_start_time()

Returns: (int) Start time of the workload generator

prepare(vms)

Sets the configuration and saves locally and on each VM in ‘vms’.

Parameters:vms (list of curie.vm.Vm) – List of VMs to prepare.
Raises:CurieTestException
run_max_concurrent_workloads(vms, max_running, runtime_secs=None, timeout_secs=None)

Runs a workload on the VMs with only the max_running number of VMs doing the workload at any point.

The list of VMs, ‘vms’ is scrambled to achieve a more even distribution of VMs performing workload across the cluster, instead of having hot nodes.

Parameters:
  • vms – (list of curie.vm.Vm) VMs upon which to run IO.
  • max_running – (int) Maximum number of IO workloads to run at once across all VMs.
  • runtime_secs – (int) Amount of time to run IO. If runtime_secs is not specified, the workload will complete when another limit is reached (e.g. the entire disk is filled).
  • timeout_secs – (int) How long to wait for the command. If the IOGen has a specified runtime, will wait timeout_secs from the time wait was called. If the IOGen has a specified runtime, will wait timeout_secs past the end of the running workload.
start(vms, runtime_secs=None, stagger_secs=None, seed=None)

Starts the workload generator on the VM for a given ‘runtime_secs’ if specified.

Parameters:
  • vms – (list of curie.vm.Vm) VMs to start workload on.
  • runtime_secs – (int) how long to run the workload.
  • stagger_secs – (int) Maximum amount of time in which to evenly space the start of the workloads.
  • seed – (str) seed value to be used by the workload generator.
Raises:

Will raise CurieException from execute_async()
stop(vms)

Stops the workload generator on the VM.

Parameters:vms – (list of curie.vm.Vm) VMs to stop workload on.
Returns:(list of booleans) True if stop succeeded for each VM.
wait(vms, timeout_secs=None, poll_secs=None)

Waits for the workload generator to complete.

Parameters:
  • vms – (list of curie.vm.Vm) List of VMs that have been started previously.
  • timeout_secs – (int) How long to wait for the command. If the IOGen has a specified runtime, will wait timeout_secs from the time wait was called. If the IOGen has a specified runtime, will wait timeout_secs past the end of the running workload.
  • poll_secs (int) – Polling interval for checking when workloads are finished.
Returns:

(list) list of named ResultResponse of (success, vm)
Return type:

results
class curie.iogen.iogen.ResultResponse(success, vm, failure_reason)

Bases: tuple

failure_reason

Alias for field number 2

success

Alias for field number 0

vm

Alias for field number 1

Module contents