The HTTP service and remote workers#
DMDO.service runs DMDO as a service: clients create and drive sessions over HTTP, follow their progress as a
live event stream, and the subproblem solves are distributed to remote workers, each pinned to the machine that
holds its analysis code. It uses only the Python standard library and is not imported by import DMDO.
When to use it#
the analyses live on different machines (a CFD code on a cluster, a structural model on a licensed workstation);
the driver is not Python, or runs in another process or container;
several people or tools need to follow or steer the same run.
For everything else, Session in-process is simpler.
Starting the service and the workers#
$ dmdo serve --host 0.0.0.0 --port 8400 --token <secret>
$ dmdo worker --connect http://dmdo-host:8400 --tags aero --token <secret> --path /opt/aero_models
$ dmdo worker --connect http://dmdo-host:8400 --tags structures --token <secret>
python -m DMDO.service serve and python -m DMDO.service worker are equivalent. --path adds a directory
to the worker’s import path, where the problem’s analysis modules are. Without --host the service listens on
127.0.0.1 only; it refuses to listen on any other interface without a token.
Driving a session from Python#
from DMDO.service import Client
client = Client("http://dmdo-host:8400", token="<secret>")
session = client.create_session(
setup_file="problem.yaml", # a path the service can read, or setup={...} (a dict)
mdo_name="wing",
blackboxes={"A1": "my_pkg.aero:A1", # import paths, resolved by the service and the workers
"A2": "my_pkg.structures:A2"},
sweep="jacobi",
placement={1: "aero", 2: "structures"}, # subproblem index -> worker tag
)
session.run() # runs in the service; returns immediately
for event in session.events(): # iteration_end records, then run_end
print(event)
print(session.result())
session.step() runs a single outer iteration instead and returns its record; session.stop() ends the run
after the current iteration; session.history() returns all iteration records.
Driving it from any language#
The service speaks plain HTTP and JSON (send Authorization: Bearer <token> with every request):
$ curl -H "Authorization: Bearer $T" -H "Content-Type: application/json" \
-d '{"setup_file": "problem.yaml", "mdo_name": "wing",
"blackboxes": {"A1": "my_pkg.aero:A1", "A2": "my_pkg.structures:A2"},
"options": {"sweep": "jacobi", "placement": {"1": "aero", "2": "structures"}}}' \
http://dmdo-host:8400/sessions
{"session_id": "3f9c2a71b0de", "status": "created", ...}
$ curl -X POST -H "Authorization: Bearer $T" http://dmdo-host:8400/sessions/3f9c2a71b0de/step
$ curl -N -H "Authorization: Bearer $T" http://dmdo-host:8400/sessions/3f9c2a71b0de/events
Method |
Path |
Action |
|---|---|---|
GET |
|
service status, number of sessions and job counters |
GET / POST |
|
list sessions / create one |
GET / DELETE |
|
status / remove an idle or finished session |
POST |
|
run to the end in the background, run one outer iteration, stop after the current one |
GET |
|
the |
GET |
|
server-sent events: |
GET / POST |
|
used by workers to claim jobs and post their results |
A complete run on one machine#
examples/service_demo.py starts a service, two worker processes tagged aero and structures, and a
client that runs the basic problem with each subproblem pinned to one worker:
import os
import secrets
import subprocess
import sys
from DMDO import load_setup
from DMDO.service import Client, start_service
HERE = os.path.dirname(os.path.abspath(__file__))
def main():
token = secrets.token_urlsafe(16)
service = start_service(root_dir=os.path.join(HERE, "_runs", "service"), token=token)
print(f"service at {service.url}")
# Two workers in their own processes; --path makes basic_disciplines importable for them.
workers = [subprocess.Popen([sys.executable, "-m", "DMDO.service", "worker", "--connect", service.url,
"--tags", tag, "--token", token, "--path", HERE, "--name", tag])
for tag in ("aero", "structures")]
try:
client = Client(service.url, token=token)
setup = load_setup(os.path.join(HERE, "basic_problem.yaml"))
setup["coord"]["c1"]["budget"] = 8
setup["MDO"]["display"] = False
session = client.create_session(
setup=setup, mdo_name="demo",
blackboxes={"A1": "basic_disciplines:A1", "A2": "basic_disciplines:A2",
"opt1": "basic_disciplines:opt1", "opt2": "basic_disciplines:opt2"},
sweep="jacobi", placement={1: "aero", 2: "structures"}, timeout=300)
session.run() # runs in the service; solves go to the workers
for event in session.events(): # live progress (server-sent events)
if event["event"] == "iteration_end":
d = event["data"]
print(f"iteration {d['iteration']}: objective {d['objective']:.6f} max|q| {d['max_inconsistency']:.3e}")
else:
print(event["event"], event["data"])
result = session.result()
print("status:", result["status"], "| variables:", result["variables"])
finally:
for w in workers:
w.terminate()
service.shutdown()
if __name__ == "__main__":
main()
Output:
DMDO worker connected to http://127.0.0.1:63514 (tags: structures)
DMDO worker connected to http://127.0.0.1:63514 (tags: aero)
service at http://127.0.0.1:63514
iteration 0: objective 3.000000 max|q| 1.100e-01
iteration 1: objective 3.864116 max|q| 7.967e-02
iteration 2: objective 4.289449 max|q| 5.994e-02
iteration 3: objective 4.136221 max|q| 8.918e-02
iteration 4: objective 3.065328 max|q| 1.254e-01
iteration 5: objective 3.071269 max|q| 8.791e-02
iteration 6: objective 3.548070 max|q| 6.247e-02
iteration 7: objective 4.337015 max|q| 4.121e-02
run_end {'stop_reason': 'Iteration budget exhausted', 'iterations': 8}
status: finished | variables: {'u_1': 1.4966531250000001, 'v_1': 0.7853000000000002, 'a_1': 0.3115753094415026,
'b_1': 1.7434863644393819, 'u_2': 1.8410000000000006, 'w_2': 7.835630000000001, 'a_2': 0.6735099999999999,
'b_2': 2.155564158811199}
The design is identical to the one computed in process (The Session API) and in a local process pool (Serial, parallel and custom execution): with the Jacobi sweep, where the subproblems run does not change the result.
Reliability#
A worker that dies in the middle of a job can be covered with
dmdo serve --lease 600: a job whose result has not arrived after 600 s is offered to another worker.A job that raises on the worker is reported back with its traceback, and the session fails with that error.
A service restart loses the sessions held in memory, but every run still writes its checkpoint (
coord_chk.pkl); resume it withSession.from_checkpoint.
Security#
Jobs and results are pickled, so the service, its workers and every client that holds the token must trust each
other: anyone who can submit a job can run code on the workers. Keep the service inside a trusted network, always
set a token when it listens on a non-loopback interface, and put a TLS-terminating reverse proxy in front of it when
traffic leaves that network. Problem files given as setup_file are read from the service’s file system.