API Reference¶

Exception hierarchy for kunene.

All errors raised by kunene derive from KuneneError, so callers can catch any workflow failure with a single except KuneneError. Previously these conditions called exit(), which raises SystemExit: that kills notebooks and long-running processes (e.g. the gRPC server in remote_actions, whose handler catches Exception but not SystemExit).

exception kunene.errors.ActionNameError¶: An action name is invalid, duplicated, or clashes with a variable name.

exception kunene.errors.AsyncActionError¶: An action running asynchronously in a child process failed.

exception kunene.errors.DataNotFoundError¶: Requested result data (e.g. a d3plot component or node id) is not available.

exception kunene.errors.EvaluationError¶: A MathEvaluation expression could not be evaluated.

exception kunene.errors.KuneneError¶: Base class for all kunene errors.

exception kunene.errors.MissingPathError¶: A required file or directory was not found.

exception kunene.errors.ParameterError¶: A parameter/variable is missing, has no value, or cannot be resolved.

exception kunene.errors.SerializationError¶: A value cannot be encoded for (or decoded from) the remote connection.

exception kunene.errors.SolverError¶: An external solver run (LS-DYNA, OpenRadioss, OpenFOAM) failed.

exception kunene.errors.SpawnError¶: A child process could not be started under the spawn start method, because something it needs is defined in the calling script.

class kunene.actions.MathEvaluation(name, cmd=None, copy_paths=None, lower_bound=None, upper_bound=None, description=None, data_type=<EvalType.NOT_SPECIFIED: 1>, keep=None)¶

Mathematical operation on results.

The expression is evaluated against a flattened view of val_dict: outputs of actions nested inside a WorkArea or sub-graph can be referenced directly by their action name.

Parameters:

name (str)
cmd (str)

Returns:

outcome of operation

Return type:

Any

solve(val_dict=None)¶

Solve/compute for action or graph. An action will return any computed results.

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Parameters:: val_dict (dict) – variable values and input of any type.
Returns:: dict with all results and inputs
Return type:: dict

class kunene.actions.WorkAction(name, cmd=None, copy_paths=None, lower_bound=None, upper_bound=None, description=None, data_type=<EvalType.NOT_SPECIFIED: 1>, keep=None)¶

Base class for the nodes in the graph. Each action encapsulate an operations on the data stream.

Parameters:

name (str)
cmd (str)
copy_paths (list) – List of file and directories to be copied to work area.
lower_bound (float) – Lower bound on output value during design
upper_bound (float) – Lower bound on output value during design
keep (list) – Glob patterns of files this action produces that a work area’s cleanup must never delete (see kunene.args.Cleanup). E.g. keep=['d3plot'] on a solver action keeps the first plot when the state files go.

Returns:

outcome of operation

Return type:

Any

allow_variables_as_arguments()¶

A decorator for the __init__() method allowing you to use variables as arguments constructing this class.

The arguments to a class can be declared to be variables, e.g. Action( name=, cmd=, arg1=FloatVariable( ‘E’, 123.4 ) ) to be used as action.solve( {‘E’:3.} ) This requires that the subclass must used the decorators allow_variables_as_arguments and assign_variables_values_to_members as:

@WorkAction.allow_variables_as_arguments

def __init__( self, name, cmd=None, v=None ):: …

@WorkAction.assign_variables_values_to_members

def solve(self, val_dict=None ):: …

Child actions are created with the variables.

You cannot do computations with the variables in __init__() because the values are only set at the end.

assign_variables_values_to_members()¶: A decorator for the solve() method allowing you to use variables as arguments constructing this class.

describe_workflow(describe=False)¶

Print both the action tree and the resulting work-directory structure for this (top-level) action.

Parameters:: describe (bool) – Include each action’s description in the tree.

format_tree(describe=False)¶

Return the action graph as an ASCII tree, rooted at this action.

Call this on the top-level action (e.g. a SimulationIterator) to see the whole workflow.

Parameters:: describe (bool) – Include each action’s description.
Returns:: the rendered tree.
Return type:: str

format_work_dir()¶

Return the predicted work-directory structure as an ASCII tree.

Shows the directories and files that running this action (or workflow) creates on disk. Files that the work area’s cleanup removes again once the run has finished are marked as such.

Returns:: the rendered directory tree.
Return type:: str

outputs()¶

Returns the output type and description of this action.

Returns:: (data_type, description)
Return type:: tuple

parameters()¶

These are the parameters defined for the WorkAction and used in the solve() method. For a graph this would be the parameters used in all the children.

Returns:: List of type Variable.
Return type:: list

print_tree(describe=False)¶: Print the action graph as a tree (see format_tree()).

print_work_dir()¶: Print the predicted work-directory structure.

report_progress(fraction=None, message=None)¶

Report how far this action has got, from inside solve.

The enclosing graph gives every action a reporter before running it, so a long action can say where it is: fraction (0..1) and a short message reach the graph’s status.json, and from there a GUI, watch_run or the per-job bars of a parallel study. The solver actions do this for you by tailing the solver’s output; a hand-written action calls this itself. It is a no-op when the action runs outside a graph.

Parameters:

fraction (float) – work done, 0..1. None leaves it unknown.
message (str) – short status line, e.g. ‘step 3 of 10’.

abstractmethod solve(val_dict: dict = None) → dict¶

Solve/compute for action or graph. An action will return any computed results.

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Parameters:: val_dict (dict) – variable values and input of any type.
Returns:: dict with all results and inputs
Return type:: dict

kunene.actions.render_tree(root)¶

Render a tree as an ASCII string with ├──/└── connectors.

Parameters:: root (tuple) – A (label, children) node, where label is a string and children is a list of the same (label, children) node tuples.
Returns:: The rendered multi-line tree.
Return type:: str

kunene.actions.validate_action_name(name)¶

Ensure an action name can be used safely as a variable in expressions.

Action names become keys in the val_dict that is passed to MathEvaluation, whose solve() runs eval(cmd, None, val_dict). For a name to be referenceable there it must be a valid Python identifier (letters, digits and underscores, not starting with a digit) and must not be a Python keyword. A name such as 'm__case_1__TE all' (embedded space) would break the eval and is rejected here.

Parameters:: name (str) – the proposed action name.
Returns:: the validated name (returned for convenience).
Return type:: str

class kunene.variables.FloatVariable(name, value, upper_bound=None, lower_bound=None, description=None)¶

A variable that may only assume an float value.

Parameters:

name (str) – Name of the variable.
value (float) – (Initial) Value of the variable.
upper_bound (float) – Maximum value of the variable.
lower_bound (float) – Minimum value of the variable.

class kunene.variables.IntSetVariable(name, value, allowable=None, description=None)¶

A variable that may only assume an integer value. It has a set of allowable values.

Parameters:

name (str) – Name of the variable.
value (int) – (Initial) Value of the variable.
allowable (set) – A set of allowable integer values.

class kunene.variables.StrSetVariable(name, value, allowable=None, description=None)¶

A variable that may only assume an string value. It has a set of allowable values.

Parameters:

name (str) – Name of the variable.
value (str) – (Initial) Value of the variable.
allowable (set) – A set of allowable string values.

class kunene.variables.UnknownVariable(name, value, description=None)¶

A variable that may assume a value of unknown type.

Parameters:

name (str) – Name of the variable.
value (str) – (Initial) Value of the variable.

class kunene.variables.Variable(name, value, data_type, description=None)¶: Abstract class for variables.

class kunene.graph_actions.DirectedGraph(name, asynch=False, work_area_path=None, cleanup=None)¶

A directed graph used to determine the order of evaluation and dependencies.

Used e.g. for MDO for solvers run in parallel.

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Parameters:

name (str) – name of the graph; must be a valid Python identifier.
asynch (bool) – run the children of this graph concurrently. Every child whose parents have finished is started at once, in a child process (there is no limit on how many run together); the default False evaluates them one after the other. The flag belongs to this graph only: it is not inherited, so a graph nested in an asynch graph runs its own children serially unless it also sets asynch=True, and it parallelises the actions of one design point, never the jobs of a SimulationIterator. Three consequences of running in another process: the children’s results travel back through a multiprocessing.Manager dict and so must be picklable; the children inherit this graph’s working directory and therefore all run in it, so branches that write files need a WorkArea each (or distinct file names) to avoid overwriting one another; and a child that raises, dies or returns nothing terminates its running siblings and raises AsyncActionError. The child is forked where the platform has fork and spawned otherwise (Windows, see kunene.util.parallel); spawning also requires the action itself to be picklable, with its class in an importable module rather than in the calling script, which the child does not re-import.
work_area_path (str) – run the graph in this directory instead of the current one, by wrapping it in a WorkArea.
cleanup (Cleanup) – only meaningful together with work_area_path; passed to the work area it creates.

Returns:

Dictionary containing action_name:action_result pairs

Return type:

dict

add_action(action, parents=None)¶

Add a node to the graph.

Parameters:

action (action)
parents (list) – Creates an graph edge from the parent actions to this node. The action will wait for the parent actions to finish.

outputs()¶

Collects outputs of all child actions.

Returns:: {action_name: (data_type, description)}
Return type:: dict

parameters()¶

These are the parameters defined for the WorkAction and used in the solve() method. For a graph this would be the parameters used in all the children.

Returns:: List of type Variable.
Return type:: list

solve(val_dict=None)¶

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Returns:

Dictionary containing action_name:action_result pairs: appended to val_dict.

Return type:

dict

update(message)¶

From observer pattern. Called by actions that have finished ([action, 'Done']) or failed asynchronously ([action, 'Failed']).

Parameters:: message (any)

class kunene.graph_actions.WorkArea(graph, work_area_path=None, copy_paths=None, cleanup=None)¶

Evaluates graph in a seperate directory. Previous results in the directory are overwritten. Required files may be copied to this area.

Use SimulationIterator if you wish to have a subdirectory for each design evaluated.

Parameters:

graph (DirectedGraph) – DirectedGraph or WorkFlow
work_area_path (str) – Default is to ./{graph.name}
copy_paths (list) – List of names of file to be copied to work area.
cleanup (Cleanup) – remove bulk solver output from the work area once the graph has run. See kunene.args.Cleanup; True selects the default policy, None (the default) keeps everything. Nothing is removed if the run raises, so a failed run can still be debugged. Note that the work area is emptied at the start of every run in any case: cleanup is about what the last run leaves behind, and about work areas nested inside a SimulationIterator, where it is inherited from the iterator unless set here.

Returns:

Output from graph (it adds nothing).

Return type:

dict

outputs()¶

Returns the output type and description of this action.

Returns:: (data_type, description)
Return type:: tuple

parameters()¶

These are the parameters defined for the WorkAction and used in the solve() method. For a graph this would be the parameters used in all the children.

Returns:: List of type Variable.
Return type:: list

solve(val_dict=None)¶

Returns:: Output from graph (it adds nothing).
Return type:: dict

class kunene.graph_actions.WorkFlow(name, actions=None, work_area_path=None, cleanup=None)¶

Calls a chain of evaluations. Results get passed down the chain. Used when everything is sequential.

Parameters:

name (str)
actions (list)
work_area_path (str)
cleanup (Cleanup) – see DirectedGraph.

Returns:

Dictionary containing action_name:action_result pairs

Return type:

dict

add_action(action)¶

Adds action to workflow.

Parameters:: name (action) – Action to add

class kunene.dyna_actions.DynaAnalysis(name, cmd='ls-dyna', input_path=None, keep=None)¶

This runs an LS-DYNA simulation. It will substitute the PARAMETER values with provided values.

Parameters:

name (str)
cmd (str) – path to ls-dyna executable or command
input_path (str) – parameterized keyword file
keep (list) – glob patterns of this run’s files that a work area’s cleanup must never delete, e.g. keep=['d3plot'] to keep the first plot when the state files are removed. See kunene.args.Cleanup.

parameters()¶

Returns the variables defined in the template. The type and value of the variables are as in the LS-DYNA input deck.

Returns: list : List of Variables.

class kunene.d3plot_actions.d3plot_File(name, d3plot_rootname='d3plot')¶

This opens the d3plot file for data extractions. The subsequent d3plot read operations must be added using methods on this method.

Parameters:

name (str)
d3plot_rootname (str) – Default is ‘d3plot’.

Return type:

success (bool)

NodalField(name, *args, **kwargs)¶

Parameters:

state (int) – -1 is last
component (str)
required_part_id (int) – optional

Returns:

Nodal field data

NodalHistory(name, *args, **kwargs)¶

Parameters:

nid (int)
component (str)

Returns:

history at node

NodalValue(name, *args, **kwargs)¶

Parameters:

nid (int)
state (int)
component (str)

Returns:

value at node

solve(val_dict=None)¶

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Returns:

Dictionary containing action_name:action_result pairs: appended to val_dict.

Return type:

dict

class kunene.jinja_actions.JinjaReplace(*args, **kwargs)¶

Action that uses Jinja2 to replace parameters in a file with values.

The parameters in the file are indicated using double curly braces, e.g., ‘{{VAR1}}’. This action reads a template file, substitutes the parameters with provided values, and writes the result to an output file.

Example

A line in the input file: ‘1, 7.8000E-06, {{E}}, 0.3, {{SIG_Y}}, 0.0, 0.0, 0.0’

With values E=210.0e9 and SIG_Y=200.0e6, becomes: ‘1, 7.8000E-06, 210.0e9, 0.3, 200.0e6, 0.0, 0.0, 0.0’

Parameters:

name (str) – The name of the action.
input_file_path (str) – Path to the input file. This is the template file marked up using jinja delimiters
output_file_path (str, optional) – Path where the processed file will be written. Defaults to kunene.args.RADIOSS_DFLT_FNAME.
val_format (str, optional) – Format string for floating point values (e.g., “%10.3g”). Defaults to “%10.3g”.

parameters()¶

Returns the variables defined in the template. The type and value of the variables are unknown.

Returns: list : List of type UnknownVariable.

solve(val_dict)¶

Solve/compute for action or graph. An action will return any computed results.

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Parameters:: val_dict (dict) – variable values and input of any type.
Returns:: dict with all results and inputs
Return type:: dict

kunene.jinja_actions.logger = <Logger kunene.jinja_actions (WARNING)>¶: The file is for cases where variables / parameters are defined inside the input decks using the jinja format ‘{{VAR1}}}’.

class kunene.remote_actions.KuneneService(actions_registry=None)¶

GetAvailableActions(request, context)¶: Missing associated documentation comment in .proto file.

GetProgress(request, context)¶: Poll the progress of a running job (status.json of its work directory). The job_id is chosen by the client and sent in ActionRequest, so a second channel can poll while the unary RunAction call blocks.

RunAction(request, context)¶: Missing associated documentation comment in .proto file.

class kunene.remote_actions.NamedServerAction(port=50051, max_workers=10)¶: Alias for ServerAction supporting named graphs.

class kunene.remote_actions.RemoteAction(name, target_action_name=None, server_address=None, copy_paths=None, output_patterns=None, progress_interval=2.0)¶

Executes a registered WorkAction on a remote server via gRPC. Requires a target_action_name to execute a pre-registered action on the server.

available_actions()¶

Queries the remote server for available registered actions.

Returns:: A dictionary mapping action names to their descriptions.
Return type:: dict

solve(val_dict=None)¶

Solve/compute for action or graph. An action will return any computed results.

A graph will append any computed results to val_dict and return that. So A.solve( {‘v1’:1.2} ) may return {‘v1’:1.2, ‘A’:3.4}, where the ‘A’:3.4 was added with ‘A’ the name of the action.

Parameters:: val_dict (dict) – variable values and input of any type.
Returns:: dict with all results and inputs
Return type:: dict

class kunene.remote_actions.ServerAction(port=50051, max_workers=10)¶

Starts a gRPC server to execute actions remotely. Can be configured with pre-registered actions (graphs).

add_graph(name, graph, description='')¶: Registers a graph (WorkAction) with a name and description.

Progress reporting for long-running workflows.

Status is exchanged between the workflow process and consumers (e.g. a GUI running as a separate process) through small status.json files written into the work directories – the same channel already used for iter_variables.json and actions_output.pkl. Files are written atomically (temp file + os.replace), so a reader never sees a half-written file, and either side can start, stop or crash independently.

Writer side (used internally by DirectedGraph and SimulationIterator):

StatusReporter – owns the status.json of one directory, writes it on every state change, and keeps a heartbeat timestamp fresh from a daemon thread so a reader can tell a slow run from a dead one.
MultiReporter – reports to several of those at once, which is how a graph inside a WorkArea fills both the work area’s own file and the file of the graph that encloses it.

Reader side (for the GUI):

StatusWatcher – cheap mtime-based polling of one status file.
RunWatcher – follows a SimulationIterator results tree: the root status plus the job(s) running now.
watch_run() – blocking generator over RunWatcher for scripts; GUIs with an event loop should call RunWatcher.poll() from a timer instead.
is_alive() – heartbeat-based liveness check of a status dict.
job_fraction() – one job’s overall progress, averaged over its actions, for a caller that wants a single number (a progress bar).

Status file schema (a graph’s file; the iterator’s root file has jobs_total/jobs_done/current_job/current_jobs instead of actions – current_jobs lists the jobs running at this moment, more than one when the iterator runs with max_workers > 1, while current_job names the last job started and is kept for readers that follow a single job):

{
  "name": "Radioss_WorkFlow",
  "state": "running",              # pending|running|idle|done|failed
  "pid": 12345,
  "heartbeat_interval": 5.0,
  "started_at": 1751871242.1,      # epoch seconds
  "updated_at": 1751880093.7,
  "actions": {
    "jinja_prep": {"state": "done",    "fraction": null, "message": null},
    "rad_solver": {"state": "running", "fraction": null, "message": null}
  }
}

The fraction and message fields carry an action’s own percent-complete: solver actions fill them from their stdout files, and an action of your own does so with self.report_progress(...). They stay null for an action that does not report, or whose output cannot be parsed.

class kunene.progress.FileProgressTail(reporter, action_name, path, parse_time, t_end, t_start=0.0, interval=2.0, tail_bytes=65536, find_t_end=None, head_bytes=8192)¶

Daemon thread reporting a solver’s percent-complete while it runs.

Solvers print the current simulation time to their (redirected) stdout file, and the termination time is known from the input deck. This thread polls the tail of that file every interval seconds, extracts the latest time with parse_time (see kunene.util.solver_progress), and reports (time - t_start) / (t_end - t_start) as the action’s fraction.

Start it right before the blocking subprocess.run and stop it in a finally (so no ‘running’ write can land after the graph marks the action done):

tail = FileProgressTail( self._progress_reporter, self.name,
                         'run_file.stdout', dyna_run_time, t_end )
tail.start()
try:
    subprocess.run( ... )
finally:
    tail.stop()

start() is a no-op when the reporter is missing/inactive or the termination time is unknown – progress reporting must never break a solver run, so parse errors are swallowed too.

When the termination time is not known up front but is printed by the solver into its output (e.g. Abstrao’s # Termination time is X s header), pass t_end=None and a find_t_end callable; the thread reads the head of the file each poll until it can extract t_end, then reports normally. (It reads the head, not the tail, because such a header scrolls out of the tail window once the file grows.)

Parameters:

reporter (StatusReporter) – where to report; may be None.
action_name (str) – the action entry to update.
path (str|Path) – the solver output file to poll.
parse_time (callable) – text -> float | None, latest sim time.
t_end (float) – termination time from the deck; None disables unless find_t_end is given.
t_start (float) – start time (OpenFOAM restarts), default 0.
interval (float) – polling period in seconds.
tail_bytes (int) – how much of the file end to read per poll.
find_t_end (callable) – head_text -> float | None to discover t_end from the file head when it is not known up front.
head_bytes (int) – how much of the file start to read for it.

stop()¶: Stop polling and wait for the thread, so no further write can race with the action’s final ‘done’/’failed’ state.

class kunene.progress.MultiReporter(reporters)¶

Reports the same action states to several status files at once.

A graph inside a WorkArea has two audiences: its own status.json in the work-area directory (which a GUI, or a standalone WorkArea, follows on its own) and the status.json of the graph that encloses the work area – a job directory, typically, whose file is what the per-job progress bars read. The work area itself holds no entry there (it is pass-through, see WorkAction._progress_names), so without this the enclosing file would say nothing at all while the solver inside runs.

Only what a graph does to an action is forwarded – reporting where it is, and failing what a dead child process left running. Starting and finishing a status file belong to the reporter that owns it. active tells FileProgressTail and _RemoteProgressPoller there is somewhere to report to.

Parameters:: reporters (list) – the reporters to write to; inactive ones (and None) are dropped, and reporting is in the given order.

class kunene.progress.RunWatcher(results_root)¶

Follows a SimulationIterator results tree: the root status.json (job counts) plus the status.json of every job it points at – ‘current_jobs’ when several run at once (max_workers > 1), else the single ‘current_job’. poll() is non-blocking – drive it from a GUI timer.

poll()¶: Return a snapshot if anything changed since the last call, else None: {'root':..., 'job_name':..., 'job':..., 'jobs': {name: status}}, where ‘jobs’ holds every job being followed and ‘job_name’/’job’ the first of them.

class kunene.progress.StatusReporter(name, directory='.', heartbeat_interval=None, takeover=False)¶

Writes the status.json of one directory.

Used by DirectedGraph.solve (per-action states, in the run directory) and SimulationIterator.solve (job counts, at the results root). A write happens on every state change; between changes a daemon thread rewrites the file every heartbeat_interval seconds so updated_at stays fresh while the process is alive.

Reporting must never break a workflow: write errors are logged and swallowed, and a reporter for a directory that is already owned by another reporter in this process silently becomes a no-op.

Fork safety (asynch actions). An asynch action runs solve() in a forked child process holding a copy of this reporter. A child never touches status.json (two processes rewriting one file would race) and never uses the inherited lock (which may have been copied in a locked state): action_state detects the pid change and writes the single entry to a per-action sidecar file (.<action>.progress.json) instead. The owning process folds sidecars into status.json on every write and every heartbeat, so child fractions surface at heartbeat cadence, and deletes them when the action reaches a terminal state (or at finish).

Parameters:

name (str) – name of the graph/iterator this status describes.
directory (str|Path) – where status.json is written. Default is the current working directory (resolved immediately, so later os.chdir calls do not move the file).
heartbeat_interval (float) – seconds between heartbeat writes. Default is progress.HEARTBEAT_INTERVAL (read at call time).
takeover (bool) – claim the directory even when another reporter in this process still owns it, stopping that one instead of going quiet. A new run of a study takes its results root over this way – the previous run is over, and its reporter must not go on heartbeating into the new run’s file. The default False is what a graph wants: a graph nested in an owned directory stays a no-op and reports through the owner.

action_state(action, state, fraction=None, message=None)¶: Set one action’s state (‘pending’|’running’|’done’|’failed’).

property active¶: False when another reporter in this process owns the directory (all reporting methods are then no-ops).

fail_running(actions, message=None)¶

Mark as failed those of actions that are still running.

Used when the process that was reporting them died – an asynch work area that crashed, or one terminated because a sibling failed. Actions that already reached a terminal state keep it, and ones that never started stay ‘pending’; only what was in flight becomes ‘failed’. Sidecars are merged first, so a child’s own last word wins over this.

finish(state='done')¶: Stop the heartbeat, write the final state and release the directory so a later run can own it again.

start(actions=None, **fields)¶

Mark the run as started and begin the heartbeat.

Parameters:

actions (list) – names of the actions to report on; all start in state ‘pending’. Omit for an iterator-level status.
fields – extra top-level fields (e.g. jobs_total=48).

update(**fields)¶: Update top-level fields (e.g. state=’idle’, jobs_done=3).

class kunene.progress.StatusWatcher(path)¶

Cheap polling reader of one status file, for a consumer in another process. poll() stats the file and re-reads it only when the mtime changed; a missing or (transiently) unparsable file is ‘no news’, never an error. The last successfully parsed status stays available as .last.

poll()¶: Return the parsed status dict if it changed since the last call, else None.

kunene.progress.format_status(snapshot)¶: Render a status dict (from StatusWatcher) or a run snapshot (from RunWatcher) as short human-readable text.

kunene.progress.is_alive(status, grace=3.0)¶: True if the process that wrote this status dict appears alive: the file was updated within grace * heartbeat_interval seconds. Use it to distinguish a slow run from a dead one; a ‘done’/’failed’ state is final regardless of the heartbeat.

kunene.progress.job_fraction(status)¶

How far one job’s graph has got, as (fraction, message).

A graph’s status file holds per-action states, and for solver actions a fraction of that action. Averaging over the actions – finished ones count as 1, the running one adds its own fraction when it has one – makes that a single number for the job, which is what a per-job progress bar needs. Returns (None, None) when the file says nothing yet.

The fraction is therefore the job’s, while the message belongs to the action running now. So that the two cannot be read as the same thing, the message says which action of how many is running and how far that action itself has got:

rad 1 of 3: time 80 of 100 (80%)

– a solver 80% through the first of three actions, which leaves the job, and the bar, at 27%. The count is the action’s place in the graph’s action list, which is the order they appear in the status file.

An asynch graph runs several actions at once, and then the message names them all with their own percentages instead:

3 of 5 running: rad_a (80%), rad_b (34%), post

kunene.progress.mark_failed(status_path, message=None)¶

Stamp a status file whose writer has died.

A process that is terminated – a job of a parallel sweep cut short because another job failed – cannot report itself, and would leave a file claiming to be running for ever. The parent that killed it calls this instead: the run state becomes ‘failed’, and so does every action that was still running. Actions that never started stay ‘pending’, since they never ran, and a file already reporting a terminal state is left as it is.

A missing or unreadable file is not an error (the job may have died before writing one), and neither is a failed write: like every other write here, reporting must never break a workflow.

Parameters:

status_path (str|Path) – the status.json to stamp.
message (str) – reason, recorded on the actions it fails.

kunene.progress.watch_run(results_root, interval=1.0)¶

Blocking generator yielding RunWatcher snapshots whenever something changed. Convenient for scripts and terminals:

for snap in watch_run('Radioss_WorkFlow'):
    print(format_status(snap))

A GUI with an event loop should use RunWatcher.poll() directly.

Design studies: evaluating a graph once per design point, and finding those results again afterwards.

SimulationIterator runs the graph in a numbered job directory per design point. JobIndex indexes those directories, which is what makes past runs retrievable: it maps each job to the variable values it was run with and to the group labels it carries.

The two belong in one module because they describe the same thing from two sides - the iterator writes the results directory, the index reads it - and because the index means nothing without the layout the iterator creates.

Each design evaluation writes its variable values to iter_variables.json and its action outputs to actions_output.pkl inside its own job_N directory. Those files describe one job but say nothing about the set: answering “which job used K=0.2?” or “which jobs belong to the baseline study?” meant walking every directory and reading every file.

jobs_index.json at the results root records one entry per job:

{ "jobs": [ { "job": "job_0",
              "groups": [ "baseline" ],
              "variables": { "K": 0.2, "T": 75 },
              "state": "done",
              "created_at": 1753,  "updated_at": 1755 } ] }

The index is a cache, never the authority: JobIndex.rebuild derives it from the job directories themselves, so result trees produced before this file existed (and trees whose index was deleted) keep working. Only the group labels are unique to the index - they are chosen by the caller and cannot be recovered from the job directories - so a rebuild preserves the labels of jobs it already knows.

Directory names stay job_0 ... job_N. Encoding variable values in the name would have to invent a float format, would break when a study adds a variable, and could not express group membership at all.

class kunene.simulation_iterator.JobIndex(root, job_prefix='job_')¶

Reads and writes the jobs_index.json of one results root.

Parameters:

root (str|Path) – the results root - the directory holding the job_N directories (a SimulationIterator’s work_area_path).
job_prefix (str) – directory name prefix, SimulationIterator.JNAME.

add_groups(jobs, groups)¶

Add group labels to the given jobs (names or records).

Returns the list of job names that were changed.

find(where=None, groups=None, state='done', match_all_groups=False, rtol=1e-09, atol=1e-12)¶

Records matching a partial set of variable values and/or groups.

Parameters:

where (dict) – variable values that must match. Only the given variables are compared, so {'K': 0.2} matches every job run with that K whatever else varied.
groups (str|list) – keep jobs carrying any of these labels (all of them when match_all_groups).
state (str) – required job state, None to accept any. Defaults to ‘done’ - a job that failed or is still running has no results to retrieve.

Returns:

list of records, in job order.

find_exact(variables, state='done', rtol=1e-09, atol=1e-12)¶

The record of the job run with exactly this set of variable values (same names, matching values), or None.

The same design point can legitimately be run more than once (a changed deck, a new solver version), so the most recent matching job wins - it describes the current state of that design point.

Used for reuse: a job that differs in a variable not mentioned in variables is a different design point and must not be reused.

group_names()¶: All group labels in use, sorted.

iter_job_dirs()¶: Job directories on disk, in numeric order.

job_path(job)¶: Absolute path of a job directory, given its name or record.

load(rebuild_if_missing=True)¶: Read the index from disk. When no index file exists but job directories do, derive one from them.

next_job_number()¶: The first unused job number, considering both the index and the directories on disk (either may be ahead of the other).

read_outputs(job)¶: Unpickle the action outputs of one job (name or record).

rebuild(save=True)¶

Derive the index from the job directories on disk.

Group labels of jobs already in the index are preserved - they exist nowhere else.

record_job(job, variables, groups=None, state='running')¶: Insert or update the entry for one job.

remove_groups(jobs, groups)¶: Remove group labels from the given jobs. Returns the job names that were changed.

save()¶: Write the index atomically, so a reader polling the file never sees it half-written.

set_state(job, state)¶: Set the state (‘running’, ‘done’, ‘failed’) of one job.

class kunene.simulation_iterator.SimulationIterator(graph, parameter_list=None, work_area_path=None, copy_paths=None, clean_start=False, groups=None, reuse_existing=False, cleanup=None, max_workers=1)¶

Used to evaluate different designs in different directories. It calls the graph in different subdirectories – a subdirectory per design. Use WorkArea to overwrite the results in a directory.

This is designed as a top-level action.

Every job is recorded in a jobs_index.json at the results root (see JobIndex in this module), which maps the job directories to the variable values they were run with and to the group labels they carry. That index backs the retrieval methods (find_jobs, results_for, collect) and reuse_existing, and can be rebuilt from the job directories at any time.

An existing results directory is added to: jobs are numbered after the ones already there, so a study can be extended in a later session and a finished job is never written over. Pass clean_start=True to delete the results directory first.

Parameters:

graph (DirectedGraph) – DirectedGraph or WorkFlow
parameter_list (list) – Only needed to provided default values to eval. Maybe not needed.
work_area_path (str) – Default is to ./{graph.name}
copy_paths (list)
clean_start (bool) – delete the results directory (jobs, index and all) before starting.
cleanup (Cleanup) – remove bulk solver output from each job directory once that job’s graph has run, so a long study does not fill the disk with field output. See kunene.args.Cleanup; True selects the default policy, None (the default) keeps every file. A job that failed is never cleaned – its deck and solver log are what you debug it with – and neither are actions_output.pkl, iter_variables.json or the index, so results_for, collect and reuse_existing keep working on a cleaned study. A WorkArea nested in the graph inherits this policy unless it sets its own.
groups (str|list) – default group label(s) for the jobs this iterator runs. Overridden per call by the groups argument of solve/collect_for_expdes/collect_for_varrange, and settable at any time as iterator.groups.
reuse_existing (bool) – when True, a design point that already has a completed job in the results directory is not run again: its stored outputs are returned instead. Default False, which runs every design point given, whether or not an equivalent job is already there.
max_workers (int) – how many jobs of a sweep may run at the same time, each in a child process of its own (forked where the platform has fork, spawned on Windows – see kunene.util.parallel). The default 1 runs them one after the other. Only the sweep methods (collect_for_expdes, collect_for_varrange) fan out; solve is one design point and always runs here. This process stays the only one that allocates job directories and writes the index, so the numbering cannot race, and a child leaves its results in its own job directory, where they are read back once it exits – so the results are structured exactly as in a serial run. A job that fails aborts the sweep, as it does when running serially: the jobs still running are terminated and AsyncActionError is raised with the child’s traceback. Set it no higher than the machine can run solvers: each job is a full graph, and a solver action may itself use several cores (and an asynch graph several processes). On a platform without fork (Windows) the child is a fresh interpreter, so the graph and its actions must be picklable and action classes must live in an importable module rather than in the calling script (SpawnError otherwise); the child does not re-import the script.

Returns:

Output from graph (it adds nothing).

Return type:

dict

add_groups(groups, jobs=None, where=None, state='done')¶

Label jobs with one or more groups.

Give either jobs (names or Paths) or where (variable values selecting the jobs). Grouping is metadata, so jobs can be labelled long after they ran, and a job can be in several groups.

Returns:: names of the jobs that were changed.
Return type:: list

collect(groups=None, where=None, state='done', match_all_groups=False)¶

Read back the results of a set of jobs already on disk.

Returns the same pair as collect_for_expdes - so a group of runs can be plotted or post-processed exactly like a fresh sweep - but nothing is executed.

Parameters:

groups (str|list) – keep jobs carrying any of these labels.
where (dict) – variable values to match (partial).
match_all_groups (bool) – require every label instead of any.

Returns:

variable name -> array of values, one per job. outcome (dict): action name -> list of values, one per job.

Return type:

par_val_dict (dict)

collect_for_expdes(exp_des, var_names, dependent_pars=None, groups=None, progress_bar=None)¶

Evaluate every design point of an experimental design, one at a time or max_workers at a time (see the constructor). The results are returned in the order the design points were given, whichever way they ran.

Parameters:

exp_des
groups (str|list) – group label(s) for the jobs of this sweep, overriding the iterator’s default.
progress_bar (bool) – passed to solve_parallel when the design points run in parallel; ignored for a serial sweep, which has no bars. The default None shows them when tqdm is installed and stderr is a terminal.

Returns:

parameter names, value list outcome (dict): evaluation name, value list. Value can be list or dict (of lists).

Return type:

par_val_dict (dict)

collect_for_varrange(var_range_dict, dependent_pars=None, groups=None, progress_bar=None)¶

Creates a combination of var_range_dict. Input is not a experimental design.

The combinations are evaluated one at a time, or max_workers at a time when the iterator was given that argument.

Parameters:

var_range_dict (dict) – variable name, values to combine.
dependent_pars (dict) – variable name, expression.
groups (str|list) – group label(s) for the jobs of this sweep, overriding the iterator’s default.
progress_bar (bool) – passed to solve_parallel when the design points run in parallel; ignored for a serial sweep, which has no bars. The default None shows them when tqdm is installed and stderr is a terminal.

Returns:

parameter names, value list outcome (dict): evaluation name, value list. Value can be list or dict (of lists).

Return type:

par_val_dict (dict)

find_job(where=None, groups=None, state='done')¶: Path of the first matching job directory, or None.

find_jobs(where=None, groups=None, state='done', match_all_groups=False)¶

Paths of the job directories matching variable values and/or groups.

Parameters:

where (dict) – variable values to match; only the variables given are compared ({'K': 0.2} matches any T).
groups (str|list) – keep jobs carrying any of these labels (all of them when match_all_groups).
state (str) – required job state, default ‘done’; None for any.

gather_outputs()¶

The action outputs of every completed job in the results directory, in job order.

Taken from the job index, so a gap in the numbering (a job directory deleted or archived) no longer cuts the list short, and jobs that failed or are still running are skipped instead of raising.

Returns:: one {action_name: value} dict per completed job.
Return type:: list

group_names()¶: All group labels in use in this results directory.

groups_of(job)¶: The group labels of a job (job name or Path).

iterdir()¶

The run directories of jobs that were previously run, in job order.

Every job directory present in the results directory is returned, including after a gap in the numbering; the former scan stopped at the first missing number. Use find_jobs to select by variable value, group or state.

Returns:: Path instances.
Return type:: list

job_index(rebuild=False)¶

The JobIndex of the results directory.

Parameters:: rebuild (bool) – re-derive the index from the job directories on disk. Group labels of known jobs are preserved.

static outcomes_as_lists(list_of_evals)¶

Transform list of evals (a list of dictionaries) to a dictionary of lists. [{‘a’:1},{‘a’:1}] -> {‘a’:[1,2]}

Parameters:: list_of_evals – a list of dictionaries
Returns:: dictionaries of lists
Return type:: dict_lists

outputs()¶

Returns the output type and description of this action.

Returns:: (data_type, description)
Return type:: tuple

parameters()¶

These are the parameters defined for the WorkAction and used in the solve() method. For a graph this would be the parameters used in all the children.

Returns:: List of type Variable.
Return type:: list

read_outputs()¶: Called in run subdirectory.

remove_groups(groups, jobs=None, where=None, state='done')¶: Remove group labels from jobs. See add_groups.

results_for(variables, groups=None)¶

The stored action outputs of the job run with these variable values - the results of a past run, without running the graph.

A job whose variables are exactly variables is preferred - the most recent one, if the design point was run more than once; a partial match is accepted when it is unambiguous.

Raises:: DataNotFoundError – when no job matches, or when several jobs match only partially. Use find_jobs or collect to get every job for a design point that was run repeatedly.

solve(val_dict=None, groups=None)¶

Evaluate one design point, here in this process. A sweep runs several at a time when max_workers > 1; this does not.

Parameters:

val_dict (dict) – variable values for this design point.
groups (str|list) – group label(s) for this job, overriding the iterator’s default.

Returns:

Output from graph (it adds nothing).

Return type:

dict

solve_parallel(design_points, groups=None, progress_bar=None)¶

Evaluate a batch of design points max_workers at a time, one child process per job – the plural of solve, which evaluates one design point here in this process.

Used by collect_for_expdes, and directly when the design points come from something that hands them out in batches (a generation of an optimizer, say) and the outputs are wanted as they are, without the design matrix collect_for_expdes builds.

A child runs the graph in the directory handed to it and leaves its results there (actions_output.pkl); this process reads them back once the child has exited, so nothing has to survive a pipe and the results are structured exactly as in a serial run. Job directories and the index stay the business of this process alone.

A job that fails aborts the batch, as it does when the sweep runs serially: the jobs still running are terminated – their results could not be used anyway – marked failed in the index, and AsyncActionError is raised carrying the child’s traceback. The jobs that finished keep their results, so the batch can be resumed with reuse_existing=True.

The children are forked where the platform has fork and spawned otherwise (Windows); kunene.util.parallel chooses. Spawning costs the graph having to be picklable – with its action classes in an importable module, since the child does not re-import the calling script – but the sweep is otherwise the same either way.

Parameters:

design_points (list) – one {variable name: value} dict per design point, as solve takes for a single one. Missing parameters are filled in with their default values.
groups (str|list) – group label(s) for these jobs, overriding the iterator’s default.
progress_bar (bool) – report progress as tqdm bars on stderr: one counting the jobs of the batch, and under it a bar per job running right now, fed from that job’s status.json so it follows the job’s actions and a solver’s percent-complete. The default None shows them when tqdm is installed and stderr is a terminal; True insists (and warns when tqdm is missing), False never shows them. The bars are a convenience for a terminal: status.json is written whichever way this is set.

Returns:

one {action name: value} dict per design point, in the order the design points were given.

Return type:

list

Raises:

ParameterError – if the batch is not a sequence of design points – notably when a single {variable name: value} dict is handed over, as solve takes.

variables_of(job)¶: The variable values a job was run with (job name or Path).

write_outputs(evals)¶: Called in run subdirectory.

kunene.simulation_iterator.as_jsonable(value)¶: Convert numpy scalars/arrays to plain Python so a value can be written to JSON. Variable values commonly come from np.arange, whose items are np.float64 and are not JSON serializable.

kunene.simulation_iterator.normalise_groups(groups)¶: Accept None, a single label or an iterable of labels and return a list of unique labels in the order given.

kunene.simulation_iterator.values_match(a, b, rtol=1e-09, atol=1e-12)¶

True when two variable values denote the same design point.

Numbers compare with a tolerance (a value written to JSON and read back is not always bit-identical); everything else compares equal. Booleans are excluded from the numeric path so True does not match 1.

Removing bulk solver output from run directories after a graph has run.

A design study keeps every job directory, and solver field output (d3plot files, OpenRadioss animation files, VTK directories) is what fills a disk. The policy for removing it is a kunene.args.Cleanup, passed as the cleanup argument of a WorkArea or a SimulationIterator.

The split of responsibility is deliberate:

the actions know which of their files are bulk output – they declare it through WorkAction._disposable_files(), and the user can subtract from that per action with DynaAnalysis( ..., keep=['d3plot'] );
the work areas know when deleting is safe – only once the whole graph has run, so a d3plot reader downstream of the solver has already read what it needed, and never for a job that failed, whose deck and log are what you need to debug it.

This module holds the walk that turns an action tree plus a policy into a per-directory delete plan, and the code that applies it.

kunene.cleanup.clean_run_dir(action, run_dir, cleanup)¶

Build and apply a cleanup plan for one run directory.

Parameters:

action (WorkAction) – the action (usually a graph) that ran.
run_dir (str|Path) – the directory it ran in.
cleanup (Cleanup) – policy, or None to do nothing.

Returns:

the paths removed, as strings.

Return type:

list

kunene.cleanup.marks_removed(name, cleanup, action)¶

Whether the work-directory display should mark an entry as removed.

Used by print_work_dir() so the predicted directory structure stays honest about what cleanup takes away again.

Parameters:

name (str) – file or directory name the action produces.
cleanup (Cleanup) – the policy in force, or None.
action (WorkAction) – the action that produces it.

Returns:

bool

class kunene.args.Cleanup(remove='bulk', keep=None, dry_run=False)¶

Policy for removing files from a run directory once the graph has run.

A study that keeps every solver’s field output fills a disk quickly. Pass a Cleanup as the cleanup argument of WorkArea or SimulationIterator to have the bulky files removed after the run, keeping the ones you still want.

The actions themselves declare which of their files are bulk output (WorkAction._disposable_files()), so remove normally needs no file names: keep subtracts from what those declarations select.

Parameters:

remove –
what is eligible for deletion.
- Cleanup.BULK ('bulk', the default) – the field output the actions declare as disposable (d3plot files, OpenRadioss animation files, VTK directories, …). Decks, logs and small history files are not touched.
- Cleanup.ALL ('*') – everything in the run directories except the protected files and keep. A job directory also holds files nothing declared (copied-in inputs, solver scratch), so this deletes more than kunene knows about; use it deliberately.
- a list of glob patterns (or a single pattern string) – exactly those, in every run directory.
keep (list) – glob patterns that are never deleted. Always wins over remove. E.g. keep=['d3plot'] drops the state files but leaves the first plot behind.
dry_run (bool) – log what would be deleted and delete nothing. Worth doing once for a new remove='*' policy.

Example:

SimulationIterator( graph, cleanup=Cleanup( keep=['d3plot'] ) )

classmethod coerce(value)¶

Turn the cleanup argument of a work area into a Cleanup.

Accepts None/False (no cleanup), True (the default bulk policy), a pattern or list of patterns, or a Cleanup.

Returns:: Cleanup or None

globs_for(action)¶

Glob patterns this policy deletes for one action.

Parameters:: action (WorkAction) – the action whose run directory is being cleaned.
Returns:: glob patterns, relative to the action’s run directory.
Return type:: list