loki.batch.scheduler

Classes

Scheduler(paths[, config, seed_routines, ...])

Work queue manager to discover and capture dependencies for a given call tree, and apply transformations for batch processing
class Scheduler(paths, config=None, seed_routines=None, preprocess=False, includes=None, defines=None, definitions=None, xmods=None, omni_includes=None, full_parse=True, frontend=Frontend.FP)

Bases: object

Work queue manager to discover and capture dependencies for a given call tree, and apply transformations for batch processing

Using a given list of paths and seed_routines (can be inferred from config), all scopes and symbols defined in the source tree are discovered and a dependency graph is created. The nodes of the dependency graph are Item objects, each corresponding to a specific Loki IR node.

The dependency graph is re-generated on-the-fly and can be filtered for specific dependency classes during traversals (see SFilter). All items, are stored in the cache of the associated item_factory.

Under the hood, the Scheduler is initialised in a three-stage procedure:

  1. A discovery step, where the minimum top-level definitions (modules and procedures) are determined for every source file in the path list.

  2. A populate step, which instantiates a first SGraph dependency graph by chasing dependencies starting from the provided seed nodes.

  3. Optionally, a full parse is triggered for all Sourcefile that appear in an Item in the dependency graph.

This first two stages rely on an incremental, incomplete parsing of the source files that extract only the minimum set of symbols in each file. This is driven by the Item._parser_class and Item._depends_class attributes, which declare the minimum RegexParserClass classes to use with the REGEX frontend.

To discover dependencies, the item’s IR is concretized. This calls Scope.make_complete with the minimum set of additional parser classes (as defined in the _parser_class attribute) that are required to discover the dependencies (e.g., calls, imports). When creating the corresponding dependency’s item, the defining scope’s item (e.g., a module containing a derived type declaration) is queried for its definitions, which in turn may trigger also a concretize step with the _parser_class of all item types that are listed in the scope item’s defines_items attribute.

A Transformation can be applied across all nodes in the dependency graph using the process() method. The class properties in the transformation implementation (such as Transformation.reverse_traversal, Transformation.traverse_file_graph or Transformation.item_filter) determine, what nodes should be processed.

config

The config object describing the Scheduler’s behaviour

Type:

SchedulerConfig

full_parse

Flag to indicate a full parse of scheduler items

Type:

bool

paths

List of paths where sourcefiles are searched

Type:

list of pathlib.Path

seeds

Names of seed routines that are the root of dependency graphs

Type:

list of str

build_args

List of frontend arguments that are given to Sourcefile.from_file when performing a full parse

Type:

dict

item_factory

Instance of the factory class for Item creation and caching

Type:

ItemFactory

Parameters:

  • paths (str or list of str) – List of paths to search for automated source file detection.

  • config (dict or str, optional) – Configuration dict or path to scheduler configuration file

  • seed_routines (list of str, optional) – Names of routines from which to populate the callgraph initially. If not provided, these will be inferred from the given config.

  • preprocess (bool, optional) – Flag to trigger CPP preprocessing (by default False).

  • includes (list of str, optional) – Include paths to pass to the C-preprocessor.

  • defines (list of str, optional) – Symbol definitions to pass to the C-preprocessor.

  • definitions (list of Module, optional) – Module object(s) that may supply external type or procedure definitions.

  • xmods (str, optional) – Path to directory to find and store .xmod files when using the OMNI frontend.

  • omni_includes (list of str, optional) – Additional include paths to pass to the preprocessor run as part of the OMNI frontend parse. If set, this replaces (!) includes, otherwise omni_includes defaults to the value of includes.

  • full_parse (bool, optional) – Flag indicating whether a full parse of all sourcefiles is required. By default a full parse is executed, use this flag to suppress.

  • frontend (Frontend, optional) – Frontend to use for full parse of source files (default FP).

source_suffixes = ['.f90', '.F90', '.f', '.F']
property sgraph

Create and return the SGraph constructed from the seeds of the Scheduler.

property items

All Item objects contained in the Scheduler dependency graph.

property dependencies

All individual pairs of Item that represent a dependency and form an edge in the :any`Scheduler` call graph.

property definitions

The list of definitions that the source files in the callgraph provide

property file_graph

Alternative dependency graph based on relations between source files

Returns:

A dependency graph containing only FileItem nodes

Return type:

SGraph

rekey_item_cache()

Rebuild the mapping of item names to items in the item_factory’s cache

This is required when a Transformation renames items during processing, and is triggered automatically at the end of the process() method if the transformation has Transformation.renames_items specified.

This update also updates config entries that are affected by the renaming.

process(transformation)

Process all items in the scheduler’s graph with either a Pipeline or a single Transformation.

A single Transformation pass invokes process_transformation() individually, while a Pipeline will apply each contained transformation in turn over the full dependency graph of the scheduler.

Parameters:

transformation (Transformation or Pipeline) – The transformation or transformation pipeline to apply

process_pipeline(pipeline)

Process a given Pipeline by applying its assocaited transformations in turn.

Parameters:

transformation (Pipeline) – The transformation pipeline to apply

process_transformation(transformation)

Process all items in the scheduler’s graph

By default, the traversal is performed in topological order, which ensures that an item is processed before the items it depends upon (e.g., via a procedure call) This order can be reversed in the Transformation manifest by setting Transformation.reverse_traversal to True.

The scheduler applies the transformation to the scope corresponding to each item in the scheduler’s graph, determined by the Item.scope_ir property. For example, for a ProcedureItem, the transformation is applied to the corresponding Subroutine object.

Optionally, the traversal can be performed on a source file level only, if the transformation has set Transformation.traverse_file_graph to True. This uses the filegraph to process the dependency tree. If combined with a Transformation.item_filter, only source files with at least one object corresponding to an item of that type are processed.

Parameters:

transformation (Transformation) – The transformation to apply over the dependency tree

callgraph(path, with_file_graph=False, with_legend=False)

Generate a callgraph visualization and dump to file.

Parameters:

  • path (str or pathlib.Path) – Path to write the callgraph figure to.

  • with_filegraph (bool or str or pathlib.Path) – Visualize file dependencies in an additional file. Can be set to True or a file path to write to.

  • with_legend (bool) – Add a key/legend to the plot. Can be enabled by setting the argument to True.

write_cmake_plan(filepath, mode, buildpath, rootpath)

Generate the “plan file” for CMake

The plan file is a CMake file defining three lists:

  • LOKI_SOURCES_TO_TRANSFORM: The list of files that are processed in the dependency graph

  • LOKI_SOURCES_TO_APPEND: The list of files that are created and have to be added to the build target as part of the processing

  • LOKI_SOURCES_TO_REMOVE: The list of files that are no longer required (because they have been replaced by transformed files) and should be removed from the build target.

These lists are used by the CMake wrappers to schedule the source updates and update the source lists of the CMake target object accordingly.