Computational resources in Boulder Opal

Based on your Boulder Opal plan, you are allocated machine hours and the maximum number of machines you can spin up to run concurrent calculations. For example, with the Performance plan, you can spin up to 4 machines simultaneously, which corresponds to 4 concurrent calculations. With the Professional plan, you can spin up to 16 machines simultaneously, which corresponds to 16 concurrent calculations. The total number of machine hours consumed is the number of hours aggregated across all machines running in your environment.

In certain scenarios, your calculation may be in the queued state for an unusually long time (for example, longer than 20 minutes). Long queue times can be primarily triggered by three reasons:

• Upscaling worker machines.
• Provisioning additional virtual machine instances.
• All allowed machines for the plan are busy running calculations.

We will cover these in this topic.

Upscaling worker machines

When you submit a calculation, it will result in multiple tasks depending on the type of calculation. Boulder Opal calculations utilize multiprocessing, hence a calculation runs as quickly as possible on a single worker. By design only a single calculation is running at a time on a single worker machine to avoid any memory issues.

For upscaling, a new worker machine spins up when a task is in the queue for more than 30 seconds. For example, in the case of a calculate_optimization with optimization_count=10, the first worker machine would pick the first optimization task. And after every 30 seconds if there are outstanding tasks in the queue, a new worker machine will spin up to handle the outstanding tasks. If each optimization takes about five minutes to run, the system by the end would have spun up 10 worker machines. This is because at every 30 second interval there would always be a task outstanding in the queue. Please note that, in this scenario, it would take longer for the calculation to complete than if there were already 10 workers spun up and ready. To reduce wait time it is better to pre-provision the worker machines as described below.

Pre-provisioning worker machines

Using the qctrl.request_machines method you can choose to avoid default upscaling in your environment. With this method, you can request the number of worker machines that need to be online before you start processing your calculations.

from qctrl import Qctrl 

qctrl = Qctrl()
qctrl.request_machines(number_of_machines)

where number_of_machines is the number of machines requested to be online (with the maximum set to the number of machines allocated in your plan).

For example,

qctrl.request_machines(4)

Waiting for 4 machines to be online...
Current environment: 0 machines online, 4 machines pending.
Current environment: 1 machine online, 3 machines pending.
Current environment: 2 machines online, 2 machines pending.
Current environment: 3 machines online, 1 machine pending.
Current environment: 4 machines online, 0 machines pending.
Requested machines (4) are online.

Shutting down the environment

You are able to shut down your computing environment to preserve resources or force the cancellation of all running jobs. This action can be performed through the Boulder Opal activity monitor, where you can monitor the status of your calculations and choose to shut down all worker machines manually by clicking the shutdown button

Provisioning additional virtual machine (VM) instances

Boulder Opal worker machines are running on Amazon AWS VM instances. There may be a scenario, where underlying VM capacity has run out, hence a new VM instance has to be provisioned. This operation takes time as it requires the underlying AWS instance to be ready before the worker can be set up to execute user calculations.

All allowed machines for the plan are busy running calculations

You can also experience long queued time, if you have multiple users running calculations at the same time but your plan does not support enough machines. If you are experiencing this more regularly, please consider upgrading to a higher plan that meets your needs.