HEROS in ATOMIQ

Atomiq is perfectly suited to describe and control the realtime-hardware in the setup. However, a good deal of the hardware orchestration concerns non-realtime devices that are not directly connected to the realtime control system but rather have to be controlled through a vast variety of different interfaces. To make all these devices available in the atomiq experiments, the HEROS framework is used. The idea behind HEROS is that every hardware object becomes a software object that can be transparently accessed via the network.

To seamlessly interact with non-realtime hardware and to make atomiq a part of the ecosystem, HEROS is fully integrated into atomiq and atomiq itself is integrated into the HEROS network.

Note

HEROS is a peer-to-peer network and does not require a central server. Rather it uses UDP broadcasts to discover other participants in the network. It will thus automagically work within an unrestricted network segment. Note however that services running in a docker container can not send or receive UDP broadcasts from or to the host network. In such cases the container (HERO and or atomiq) can be configured to use the host network or, alternatively, connect to a zenoh router.

The zenoh connection is set through the HEROS environment variables. They apply to the atomiq master and to the experiment workers it starts. By default atomiq runs as a zenoh peer. The router then mainly brokers the discovery and peers talk directly when they can reach each other. If the master cannot reach the HEROs directly, as in a docker container without the host network, use client mode. All traffic then goes through the router.

environment:
  HEROS_ZENOH_MODE: client
  HEROS_ZENOH_CONNECT: tcp/<IP_OR_HOSTNAME>:7447

HEROS_DISCOVER_TIMEOUT sets how long a HERO lookup waits for an answer, in seconds. See the HEROS documentation for all variables.

Using HEROs in the Atomiq experiment

Using an available HERO in your atomiq experiment is a easy as putting it's name to your components list. Let's assume there is a dummy voltage source available as a HERO with the name rfsource_dummy in the default HEROS realm heros. By prefixing the HERO name with the magic identifier "$" you can state in your components list that the experiment requires the HERO like shown below

from atomiq import AtomiqExperiment

class HerosTest(AtomiqExperiment):
    components = ["herosink", "$rfsource_dummy"]

@kernel
def step(self, point):
    self.rfsource_dummy.set_amplitude(0.56)

Note

If your hero is not in the default realm (as set by the HEROS_REALM environment variable), you can use the syntax $realm/hero_name to reference the hero. If doing so in the components list, the hero becomes available as self.realm_hero_name.

Dashes "-" will be replaced by "_" in the attribute name. E.g. $realm/hero-name will be available as self.realm_hero_name.

Any HERO imported this way inherits from Component inside ATOMIQ. However, if the HERO specifies in it's metadata that it is compatible with a more specific class inside atomiq.components then it is based off this more specific class and atomiq assumes that the necessary interface exists in the HERO. This way, it is possible to have non-realtime devices like RF sources, voltage and current sources, ADCs, simples switches, etc. look exactly like their realtime counterparts from a software perspective. Inside atomiq, you can use all methods, attributes and events that the HERO exposes. If your HERO carries proper type annotations, you can also access the HERO from within kernel code sections (of course this then issues an rpc call into the artiq master who then does the calling of the HERO).

For more details on how to make your device a HERO and how the definition of the atomiq interfaces in HEROS work, check the documentation of HEROS and herosdevices.

If your HERO implements an ATOMIQ component interface, it can be used as any other component of that type. That allows also reference it in the component definition in the very same way. For example an AOM that is driven by an RF source which is controlled by a HERO rfsource_dummy which implements atomiq.components.electronics.rfsource.RFSource could look like the following in your components definition

{
    "aom_cooler": {
        "classname": "atomiq.components.optoelectronics.lightmodulator.AOM",
        "arguments": {
            "rfsource": "$rfsource_dummy",
            "switch": "$rfsource_dummy",
            "center_freq": 80e6,
            "bandwidth": 40e6,
            "switching_delay": 30e-9,
            "order": 1,
            "passes": 2
        }
    }
}

Atomiq becoming a HERO

Atomiq comes with it's own frontend to start the artiq master in atomiq.frontend.atomiq_master. Using this starter allows to modify artiq such that the artiq scheduler becomes itself a HERO. This allows to have a HERO for managing the execution of experiments. In turn this allows to have external logic acting on your experiment-analysis-experiment cycle. Re-scheduling an experiment with different parameters or scheduling a completely different experiment depending on the outcome of a previous experiment or the computation result of some other HERO (think AI) becomes easily possible with this feature.

Note

You can specify the realm in which all Atomiq related HEROs are running with the HEROS_REALM environment variable of the Atomiq Master.

Scheduling Experiments

In addition to the scheduler itself, the experiment database is available as a HERO upon starting the atomiq.frontend.atomiq_master. Using this database, scheduling an experiment becomes as easy as:

from heros import RemoteHERO
scheduler = RemoteHERO("atomiq-scheduler",realm="heros")
db = RemoteHERO("atomiq-experimentdb", realm="heros")

# list all experiments (the ones you would see in the dashboard)
db.list_experiments()

# generate a new run definition (internally denoted as expid in ARTIQ)
expid = db.get_experiment("examples/ATQExperiment")

# modify an argument
expid["dummy_voltage"]["repetitions"] = 10

# schedule the experiment to the "main" pipeline
rid = scheduler.submit("main", expid)
print(f"Experiment with class name {expid['class_name']} scheduled as RID {rid}")

Inspecting the Device Database

The experiment database HERO exposes the ARTIQ device database and the atomiq components defined in it. Both methods take an optional key_filter, a Unix shell style glob pattern matched against the keys.

from heros import RemoteHERO
db = RemoteHERO("atomiq-experimentdb", realm="heros")

# all ARTIQ devices
db.get_device_db()

# only the urukul devices
db.get_device_db("urukul*")

# all atomiq components
db.get_components()

# only the components whose name starts with "ttl_"
db.get_components("ttl_*")

Atomiq keeps its component definitions under the components key of the ARTIQ device database. get_device_db strips that key, so the two methods return disjoint sets. get_components returns an empty dict if the device database defines no components at all.

Duplicating Running Experiments

The run configuration (the expid) of a running experiment can be extracted via the scheduler, saved and run again. Note that this method does not save the state of the script itself, it merely saves the parameter configuration from the dashboard.

import json
from heros import RemoteHERO
scheduler = RemoteHERO("atomiq-scheduler", realm="heros")

# get the expid from the running experiment with RID 666
rid = 666
expid = scheduler.get_expid(rid)

# save it to a json-file
with open("rid_666.json", "w") as fp:
    json.dump(expid, fp, indent=4)

Retrieving the Master Log

The atomiq master keeps its last 1000 log messages in a ring buffer that the scheduler HERO exposes. This gives remote access to the same messages that show up in the log pane of the dashboard, without having to connect to the logging port of the master.

from heros import RemoteHERO
scheduler = RemoteHERO("atomiq-scheduler", realm="heros")

# all buffered entries, oldest first
scheduler.get_log()

# only the 20 most recent entries
scheduler.get_log(maxlen=20)

Each entry is a dict with the same fields the dashboard shows:

{
    "level": "INFO",
    "source": "worker(11,heros_example.py)",  # "master" for messages of the master itself
    "time": 1793889326.42,  # UNIX timestamp
    "message": "heros_example:<message>",  # <logger name>:<message>
}

The source of a worker contains the RID of the run that emitted the message. Messages without a run carry master for the master process (including the device database), worker(scan,<file>) for the repository scan and worker(examine,<file>) for argument recomputation from the dashboard. Entries older than the last 1000 messages are discarded.

Using Experiment HEROs

All scheduled and running experiments are HEROs as well. This allows you to interact with the living experiment from remote. You can readout important internal variables (step_counter, rid, etc) and request termination, or add some custom functions. These experiment HEROs also provide an event emit_data that can be used to broadcast data out of your experiment, as described in the following section.

The scheduler HERO triggers an event run_started when a new experiment enters the run phase and an event run_ended when the run phase finished.The metadata transmitted with this event contains all information to get the HERO of the just starting experiment.

The class-level attribute atomiq.AtomiqExperiment.TAGS can be used to tag the experiments. During creation of the AtomiqHERO, these tags populate heros.LocalHERO._hero_tags and can - for example - be accessed to trigger downstream actions:

class SampleSpectroscopy(AtomiqExperiment):
    TAGS = ["calibration", "analyze_spec"]
    # the rest of the experiment

Bob's machine where the analysis should run:

from heros import RemoteHERO
scheduler = RemoteHERO("atomiq-scheduler", realm="heros")

def handle_run_started(data):
    atomiq_run_hero = RemoteHERO(data["hero_name"], realm="heros")
    if "analyze_spec" in atomiq_run_hero._hero_tags:
        is_calibration = "calibration" in atomiq_run_hero._hero_tags
        # pass the run to the analysis worker
        do_analysis(atomiq_run_hero, publish=is_calibration)

scheduler.run_started.connect(handle_run_started)

Using the Experiment HERO as Data Sink

with the following in your atomiq components dict:

"herosink": {
    "classname": "atomiq.components.basics.datasink.HEROSink"
}

broadcasting your data becomes as easy as

from atomiq import AtomiqExperiment

class HerosTest(AtomiqExperiment):
    components = ["herosink"]

    @kernel
    def step(self, point):
        # emit env data (like run ids, arguments for the current step, etc.)
        self.herosink.submit_env(point)

        # emit custom data
        self.herosink.submit_data(["my_variable", "your_variable"], [30.21, -13.0])

To access the data put into the herosink, you can get a reference to the experiment HERO and connect your analysis or processing method to the event emit_data. This could look as the following:

from heros import RemoteHERO

def process_and_print(data):
    # processing...
    print(data)

experiment = RemoteHERO("atomiq-run-<rid>", realm="heros")
experiment.emit_data.connect(process_and_print)

Here, <rid> is the run ID of the experiment.

The RemoteHERO of the currently running experiment can additionally be retrieved from the scheduler:

from heros import RemoteHERO

def process_and_print(data):
    # processing...
    print(data)

scheduler = RemoteHERO("atomiq-scheduler", realm="heros")
experiment = scheduler.get_current_run()
if experiment is not None:
    experiment.emit_data.connect(process_and_print)

The data emitted through the emit_data event by submit_env contains the values of all experiment arguments (scannable and non-scannable) for the current step, one entry per argument, together with the following metadata:

Keys of the emit_data event payload submitted via submit_env

Key

Description

<argument name>

One entry per experiment argument (scannable and non-scannable) holding its value in the current step.

timestamp

Timestamp of the data point in seconds.

run_id

The RID of the current run.

run_timestamp

Unix timestamp (seconds) of the start of the run.

step_counter

Counter of the current step within the scan.

scan_argument_order

List of the names of the scanned arguments, ordered from the inner to the outer scan loop.

identifier

Unique identifier of the data point.

Using ARTIQ Datasets via the Experiment HERO

While using native HEROS data handling via the emit_data event is generally preferable, there are usecases, such as migrating an existing experiment to atomiq or relying on the ARTIQ .h5 saving mechanism, which require handling of ARTIQ datasets through HEROS.

Therefore the experiment HERO exposes the dataset functions of HasEnvironment set_dataset, mutate_dataset, append_to_dataset, get_dataset and get_dataset_metadata.

Note

The exposed functions are prefixed with _ to make it more clear that usage is discouraged.

This functions act exactly the same as when you would call them in your experiment code.

In the following example, we set a dataset test_set at the start of a run:

from heros import RemoteHERO
scheduler = RemoteHERO("atomiq-scheduler", realm="heros")

def handle_run_started(data):
    atomiq_run_hero = RemoteHERO(data["hero_name"], realm="heros")
    atomiq_run_hero._set_dataset("test_set", [2,3,4])

scheduler.run_started.connect(handle_run_started)