Creating Scenarios with YAML

Scenarios in Curie are implemented as YAML but will include other peripheral files such as those for workload descriptions or additional scripts.

Curie Scenario Structure

Scenarios must have the following structure to be valid:

/<test_name>
|── test.yml
|-- <some_file.fio>
├── <additional resources>
├── ...

Scenario resource files may be referenced within each of the steps. For example, if a workload generator is used to create storage traffic in the scenario, the workload configuration files are placed in the subdirectory (alongside test.yml), and are referenced in steps from the curie.test.steps.workload module.

A typical scenario directory structure might look like this:

/custom_mixed_database_scenario
├── test.yml
├── dss.fio
└── oltp.fio

Scenarios included with X-Ray and examples on how to create scenarios can be found in the X-Ray Scenarios Repository.

Required Fields

name

The internally used name of the scenario. This must be unique.

1
name: database_colocation_standard

display_name

The externally visible name of the scenario. This must be unique.

3
display_name: "Database Colocation: Standard Intensity"

summary

A one-line description of the scenario; A tagline.

5
summary: Impact of running colocated DB workloads (OLTP+DSS)

description

A detailed description of the scenario. This should cover the goals of the scenario, the steps performed, and how to interpret the results. Today, this section is formatted in HTML.

 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
description: |
  <strong>Test Objectives</strong><br />
  This test is designed to show the effects of running two different database
  workloads (OLTP and DSS) on three different nodes within the same cluster.
  <ol>
    <li>Standard Intensity:</li>
      <ul class="indented">
        <li>OLTP: 4000 IOPS with 28x4 GB working set per VM</li>
        <li>OLTP Log: 200 IOPS with 2x2 GB working set per VM</li>
        <li>DSS: 720 IOPS with 28x4 GB working set per VM</li>
        <li>DSS Log: 200 IOPS with 2x2 GB working set per VM</li>
      </ul>
    <li>High Intensity:</li>
      <ul class="indented">
        <li>OLTP: 4000 IOPS with 28x4 GB working set per VM</li>
        <li>OLTP Log: 200 IOPS with 2x2 GB working set per VM</li>
        <li>DSS: 1200 IOPS with 32x4 GB working set per VM</li>
        <li>DSS Log: 400 IOPS with 2x2 GB working set per VM</li>
      </ul>
  </ol>
  <Strong>Setup</strong>
  <ol>
    <li>Deploy the OLTP DB VM on node 1, and a DSS DB VM on nodes 2 and 3.</li>
    <li>Pre-fill the VMs with data.</li>
    <li>Warm up the OLTP DB for 30 minutes.</li>
  </ol>
  <strong>Measurement</strong>
  <ol>
    <li>Start the OLTP workload with a runtime of 60 minutes.</li>
    <li>Wait 30 minutes while OLTP workload runs alone.</li>
    <li>Start the DSS workload with a runtime of 30 minutes.</li>
  </ol>
  <strong>Test Requirements</strong>
  <ul class="indented">
    <li>vCPUs: 4 vCPUs on nodes 1-3</li>
    <li>RAM: 4 GB on nodes 1-3</li>
    <li>Cluster Storage: 348 GB</li>
    <li>IP Addresses: 3</li>
  </ul>

tags

A list of strings that can be used when filtering search results.

47
48
49
50
tags:
  - performance
  - network
  - mixed workloads

estimated_runtime

The estimated duration of the scenario in seconds.

52
estimated_runtime: 6000

vms

A list of VM group names, with configuration information for each. A VM group represents a group of VMs that are all identical, and perform operations at the same time.

  • Required
    • template The name of the VM image template to use.
  • Optional
    • vcpus The number of vCPUs that should be allocated for each VM in the VM group.
    • ram_mb The amount of RAM in megabytes that will be allocated for each VM in the VM group when deployed.
    • data_disks A dictionary containing count and size of the data disks that should be created for the
    • nodes A string that describes the nodes in the cluster upon which the VMs should be deployed. This string is a comma-separated series of slices described using Python’s built-in slicing behavior.
    • count_per_node The number of VMs to deploy on each of the nodes described by nodes.
    • count_per_cluster The total number of VMs to deploy evenly across the nodes described by nodes.

Note

Either count_per_node or count_per_cluster must be defined. They can not be used together.

54
55
56
57
58
59
60
61
62
63
vms:
  - OLTP:
      template: ubuntu1604
      vcpus: 4
      ram_mb: 4096
      data_disks:
        count: 6
        size: 32
      nodes: 0
      count_per_node: 1

workloads

A list of workload names, with configuration information for each. A workload is used to generate storage load on the cluster using a VM group.

  • Required
    • vm_group: The name of the VM group to run the workload. This is defined in the vms section.
    • config_file: The name of the workload configuration file to run. This is usually a .fio file in the same directory as the scenario’s YAML file.
74
75
76
77
workloads:
  - OLTP:
      vm_group: OLTP
      config_file: oltp.fio

results

A list of result names, with configuration information for each. Each result will show up as a plot in the X-Ray UI.

  • Required
    • workload_name: The name of the workload generating the data for this set of results. This is defined in the workloads section.

    • result_type: The type of plot to render for this result. Valid choices are:

      • iops: Create a line plot showing the number of I/O operations per second over time.
      • errors: Create a line plot showing the number of cumulative I/O errors over time.
      • active: Create a line plot showing I/O activity (boolean) over time.
      • latency: Create a line plot showing latency of I/O operations over time.
    • aggregate (Optional): Combine the results from all VMs associated with the workload into a single plot. Options include:

      • sum
      • min
      • mean
      • median
      • max
85
86
87
88
89
90
91
results:
  - OLTP IOPS:
      vm_group: OLTP
      result_type: iops
      report_group: performance
      report_metrics:
        - Variability

setup

A list of step names, with parameters for each. These steps are performed during the scenario’s setup phase. These are the steps that must occur before the run phase can be reached.

Because these setup steps occur before the measurement period of the scenario begins, no results are rendered in the X-Ray UI during these steps.

For a complete list of the steps available, see the curie.test.steps package.

Note

The test parameter of each step is passed in implicitly, and should not be included in the YAML.

111
112
113
114
115
116
117
118
  - vm_group.PowerOn:
      vm_group_name: OLTP
  - vm_group.PowerOn:
      vm_group_name: DSS
  - workload.PrefillRun:
      workload_name: OLTP
  - workload.PrefillRun:
      workload_name: DSS

run

A list of step names, with parameters for each. These steps are performed during the scenario’s run phase.

Results are rendered in the X-Ray UI during these steps.

For a complete list of the steps available, see the curie.test.steps package.

Note

The test parameter of each step is passed in implicitly, and should not be included in the YAML.

132
133
134
135
136
137
  - test.Wait:
      duration_secs: 1800
  - workload.Start:
      workload_name: DSS
      runtime_secs: 1800
      annotate: True

Steps

For a complete list of the steps available, see the curie.test.steps package.

Note

The test parameter of each step is passed in implicitly, and should not be included in the YAML.

Working Example

Database Colocation Scenario
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
name: database_colocation_standard

display_name: "Database Colocation: Standard Intensity"

summary: Impact of running colocated DB workloads (OLTP+DSS)

description: |
  <strong>Test Objectives</strong><br />
  This test is designed to show the effects of running two different database
  workloads (OLTP and DSS) on three different nodes within the same cluster.
  <ol>
    <li>Standard Intensity:</li>
      <ul class="indented">
        <li>OLTP: 4000 IOPS with 28x4 GB working set per VM</li>
        <li>OLTP Log: 200 IOPS with 2x2 GB working set per VM</li>
        <li>DSS: 720 IOPS with 28x4 GB working set per VM</li>
        <li>DSS Log: 200 IOPS with 2x2 GB working set per VM</li>
      </ul>
    <li>High Intensity:</li>
      <ul class="indented">
        <li>OLTP: 4000 IOPS with 28x4 GB working set per VM</li>
        <li>OLTP Log: 200 IOPS with 2x2 GB working set per VM</li>
        <li>DSS: 1200 IOPS with 32x4 GB working set per VM</li>
        <li>DSS Log: 400 IOPS with 2x2 GB working set per VM</li>
      </ul>
  </ol>
  <Strong>Setup</strong>
  <ol>
    <li>Deploy the OLTP DB VM on node 1, and a DSS DB VM on nodes 2 and 3.</li>
    <li>Pre-fill the VMs with data.</li>
    <li>Warm up the OLTP DB for 30 minutes.</li>
  </ol>
  <strong>Measurement</strong>
  <ol>
    <li>Start the OLTP workload with a runtime of 60 minutes.</li>
    <li>Wait 30 minutes while OLTP workload runs alone.</li>
    <li>Start the DSS workload with a runtime of 30 minutes.</li>
  </ol>
  <strong>Test Requirements</strong>
  <ul class="indented">
    <li>vCPUs: 4 vCPUs on nodes 1-3</li>
    <li>RAM: 4 GB on nodes 1-3</li>
    <li>Cluster Storage: 348 GB</li>
    <li>IP Addresses: 3</li>
  </ul>

tags:
  - performance
  - network
  - mixed workloads

estimated_runtime: 6000

vms:
  - OLTP:
      template: ubuntu1604
      vcpus: 4
      ram_mb: 4096
      data_disks:
        count: 6
        size: 32
      nodes: 0
      count_per_node: 1
  - DSS:
      template: ubuntu1604
      vcpus: 4
      ram_mb: 4096
      data_disks:
        count: 6
        size: 32
      nodes: 1,2
      count_per_node: 1

workloads:
  - OLTP:
      vm_group: OLTP
      config_file: oltp.fio
  - OLTP Warmup:
      vm_group: OLTP
      config_file: oltp.fio
  - DSS:
      vm_group: DSS
      config_file: dss.fio

results:
  - OLTP IOPS:
      vm_group: OLTP
      result_type: iops
      report_group: performance
      report_metrics:
        - Variability
  - OLTP I/O Latency:
      vm_group: OLTP
      result_type: latency
      report_group: performance
      report_metrics:
        - Median
  - DSS IOPS:
      vm_group: DSS
      result_type: iops
      report_group: performance
      report_metrics:
        - Variability

setup:
  - cluster.CleanUp: {}
  - vm_group.CloneFromTemplate:
      vm_group_name: OLTP
  - vm_group.CloneFromTemplate:
      vm_group_name: DSS
  - vm_group.PowerOn:
      vm_group_name: OLTP
  - vm_group.PowerOn:
      vm_group_name: DSS
  - workload.PrefillRun:
      workload_name: OLTP
  - workload.PrefillRun:
      workload_name: DSS
  - workload.Start:
      workload_name: OLTP Warmup
      runtime_secs: 1800
  # Wait for counters to stabilize before entering Run phase.
  - test.Wait:
      duration_secs: 10

run:
  - workload.Start:
      workload_name: OLTP
      runtime_secs: 3600
      stagger_secs: 900
      async: True
  - test.Wait:
      duration_secs: 1800
  - workload.Start:
      workload_name: DSS
      runtime_secs: 1800
      annotate: True
  - workload.Wait:
      workload_name: OLTP

teardown:
  - cluster.CleanUp: {}