The Pyomo Configuration System
The Pyomo configuration system provides a set of three classes
(ConfigDict, ConfigList, and
ConfigValue) for managing and documenting structured
configuration information and user input. The system is based around
the ConfigValue class, which provides storage for a single
configuration entry. ConfigValue objects can be grouped using
two containers (ConfigDict and ConfigList) that
provide functionality analogous to Python’s dict and
list classes, respectively.
At its simplest, the configuration system allows for developers to specify a
dictionary of documented configuration entries:
from pyomo.common.config import (
ConfigDict, ConfigList, ConfigValue
)
config = ConfigDict()
config.declare('filename', ConfigValue(
default=None,
domain=str,
description="Input file name",
))
config.declare("bound tolerance", ConfigValue(
default=1E-5,
domain=float,
description="Bound tolerance",
doc="Relative tolerance for bound feasibility checks"
))
config.declare("iteration limit", ConfigValue(
default=30,
domain=int,
description="Iteration limit",
doc="Number of maximum iterations in the decomposition methods"
))
Users can then provide values for those entries, and retrieve the
current values:
>>> config['filename'] = 'tmp.txt'
>>> print(config['filename'])
tmp.txt
>>> print(config['iteration limit'])
30
For convenience, ConfigDict objects support read/write access via
attributes (with spaces in the declaration names replaced by
underscores):
>>> print(config.filename)
tmp.txt
>>> print(config.iteration_limit)
30
>>> config.iteration_limit = 20
>>> print(config.iteration_limit)
20
Domain validation
All Config objects support a domain keyword that accepts a callable
object (type, function, or callable instance). The domain callable
should take a single argument (the incoming data value) and map it onto
the desired domain, optionally
performing domain validation (see ConfigValue,
ConfigDict, and ConfigList for more
information). This allows client code to accept a very flexible set of
inputs without “cluttering” the code with input validation:
>>> config.iteration_limit = 35.5
>>> print(config.iteration_limit)
35
>>> print(type(config.iteration_limit).__name__)
int
In addition to common types (like int, float,
bool, and str), the configuration system provides a
number of custom domain validators for common use cases:
Bool(val)
|
Domain validator for bool-like objects. |
Integer(val)
|
Domain validation function admitting integers |
PositiveInt(val)
|
Domain validation function admitting strictly positive integers |
NegativeInt(val)
|
Domain validation function admitting strictly negative integers |
NonNegativeInt(val)
|
Domain validation function admitting integers >= 0 |
NonPositiveInt(val)
|
Domain validation function admitting integers <= 0 |
PositiveFloat(val)
|
Domain validation function admitting strictly positive numbers |
NegativeFloat(val)
|
Domain validation function admitting strictly negative numbers |
NonPositiveFloat(val)
|
Domain validation function admitting numbers less than or equal to 0 |
NonNegativeFloat(val)
|
Domain validation function admitting numbers greater than or equal to 0 |
In(domain[, cast])
|
Domain validation class admitting a Container of possible values |
InEnum(domain)
|
Domain validation class admitting an enum value/name. |
IsInstance(*bases[, document_full_base_names])
|
Domain validator for type checking. |
ListOf(itemtype[, domain, string_lexer])
|
Domain validator for lists of a specified type |
Module([basePath, expandPath])
|
Domain validator for modules. |
Path([basePath, expandPath])
|
Domain validator for a path-like object. |
PathList([basePath, expandPath])
|
Domain validator for a list of path-like objects. |
DynamicImplicitDomain(callback)
|
Implicit domain that can return a custom domain based on the key. |
Configuring class hierarchies
A feature of the configuration system is that the core classes all implement
__call__, and can themselves be used as domain values. Beyond
providing domain verification for complex hierarchical structures, this
feature allows ConfigDict objects to cleanly support extension
and the configuration of
derived classes. Consider the following example:
>>> class Base:
... CONFIG = ConfigDict()
... CONFIG.declare('filename', ConfigValue(
... default='input.txt',
... domain=str,
... ))
... def __init__(self, **kwds):
... self.cfg = self.CONFIG(kwds)
... self.cfg.display()
...
>>> class Derived(Base):
... CONFIG = Base.CONFIG()
... CONFIG.declare('pattern', ConfigValue(
... default=None,
... domain=str,
... ))
...
>>> tmp = Base(filename='foo.txt')
filename: foo.txt
>>> tmp = Derived(pattern='.*warning')
filename: input.txt
pattern: .*warning
Here, the base class Base declares a class-level attribute CONFIG as a
ConfigDict containing a single entry (filename). The derived class
(Derived) then starts by making a copy of the base class’ CONFIG,
and then defines an additional entry (pattern). Instances of the base
class will still create cfg attributes that only have the single
filename entry, whereas instances of the derived class will have cfg
attributes with two entries: the pattern entry declared by the derived
class, and the filename entry “inherited” from the base class.
An extension of this design pattern provides a clean approach for
handling “ephemeral” instance options. Consider an interface to an
external “solver”. Our class implements a solve() method that takes a
problem and sends it to the solver along with some solver configuration
options. We would like to be able to set those options “persistently”
on instances of the interface class, but still override them
“temporarily” for individual calls to solve(). We implement this by
creating copies of the class’s configuration for both specific instances
and for use by each solve() call:
class Solver:
CONFIG = ConfigDict()
CONFIG.declare('iterlim', ConfigValue(
default=10,
domain=int,
))
def __init__(self, **kwds):
self.config = self.CONFIG(kwds)
def solve(self, model, **options):
config = self.config(options)
# Solve the model with the specified iterlim
config.display()
>>> solver = Solver()
>>> solver.solve(None)
iterlim: 10
>>> solver.config.iterlim = 20
>>> solver.solve(None)
iterlim: 20
>>> solver.solve(None, iterlim=50)
iterlim: 50
>>> solver.solve(None)
iterlim: 20
This design pattern is widely used across Pyomo; particularly for
configuring solver interfaces and transformations. We provide a
decorator to simplify the process of documenting these CONFIG
attributes:
from pyomo.common.config import document_class_CONFIG
@document_class_CONFIG(methods=['solve'])
class MySolver:
"""Interface to My Solver"""
#
#: Global class configuration; see :ref:`MySolver_CONFIG`
CONFIG = ConfigDict()
CONFIG.declare('iterlim', ConfigValue(
default=10,
domain=int,
doc="Solver iteration limit",
))
#
def __init__(self, **kwds):
#: Instance configuration; see :ref:`MySolver_CONFIG`
self.config = self.CONFIG(kwds)
#
def solve(self, model, **options):
"""Solve `model` using My Solver"""
#
config = self.config(options)
# Solve the model with the specified iterlim
config.display()
>>> print(MySolver.__doc__)
Interface to My Solver
**Class configuration**
This class leverages the Pyomo Configuration System for managing
configuration options. See the discussion on :ref:`configuring class
hierarchies <class_config>` for more information on how configuration
class attributes, instance attributes, and method keyword arguments
interact.
.. _MySolver::CONFIG:
CONFIG
------
iterlim: int, default=10
Solver iteration limit
>>> print(MySolver.solve.__doc__)
Solve `model` using My Solver
Keyword Arguments
-----------------
iterlim: int, default=10
Solver iteration limit
Interacting with argparse
In addition to basic storage and retrieval, the configuration system provides
hooks to the argparse command-line argument parsing system. Individual
configuration entries can be declared as argparse arguments using the
declare_as_argument() method. To make declaration
simpler, the declare() method returns the declared configuration
object so that the argument declaration can be done inline:
import argparse
config = ConfigDict()
config.declare('iterlim', ConfigValue(
domain=int,
default=100,
description="iteration limit",
)).declare_as_argument()
config.declare('lbfgs', ConfigValue(
domain=bool,
description="use limited memory BFGS update",
)).declare_as_argument()
config.declare('linesearch', ConfigValue(
domain=bool,
default=True,
description="use line search",
)).declare_as_argument()
config.declare('relative tolerance', ConfigValue(
domain=float,
description="relative convergence tolerance",
)).declare_as_argument('--reltol', '-r', group='Tolerances')
config.declare('absolute tolerance', ConfigValue(
domain=float,
description="absolute convergence tolerance",
)).declare_as_argument('--abstol', '-a', group='Tolerances')
The ConfigDict can then be used to initialize (or augment) an
argparse.ArgumentParser object:
parser = argparse.ArgumentParser("tester")
config.initialize_argparse(parser)
Key information from the ConfigDict is automatically transferred over
to the ArgumentParser object:
>>> print(parser.format_help())
usage: tester [-h] [--iterlim INT] [--lbfgs] [--disable-linesearch]
[--reltol FLOAT] [--abstol FLOAT]
...
-h, --help show this help message and exit
--iterlim INT iteration limit
--lbfgs use limited memory BFGS update
--disable-linesearch [DON'T] use line search
Tolerances:
--reltol... -r FLOAT relative convergence tolerance
--abstol... -a FLOAT absolute convergence tolerance
Parsed arguments can then be imported back into the ConfigDict:
>>> args=parser.parse_args(['--lbfgs', '--reltol', '0.1', '-a', '0.2'])
>>> args = config.import_argparse(args)
>>> config.display()
iterlim: 100
lbfgs: true
linesearch: true
relative tolerance: 0.1
absolute tolerance: 0.2
Accessing user-specified values
It is frequently useful to know which values a user explicitly set, and
which values a user explicitly set but have never been retrieved. The
configuration system provides two generator methods to return the items
that a user explicitly set (user_values()) and the items that
were set but never retrieved (unused_user_values()):
>>> print([val.name() for val in config.user_values()])
['lbfgs', 'relative tolerance', 'absolute tolerance']
>>> print(config.relative_tolerance)
0.1
>>> print([val.name() for val in config.unused_user_values()])
['lbfgs', 'absolute tolerance']
Outputting the current state
Configuration objects support two methods for generating output:
display() and
generate_yaml_template(). The simpler is
display(), which prints out the current values of
the configuration object (and if it is a container type, all of its
children). generate_yaml_template() is similar to
display(), but also includes the description fields
as formatted comments.
solver_config = config
config = ConfigDict()
config.declare('output', ConfigValue(
default='results.yml',
domain=str,
description='output results filename'
))
config.declare('verbose', ConfigValue(
default=0,
domain=int,
description='output verbosity',
doc='This sets the system verbosity. The default (0) only logs '
'warnings and errors. Larger integer values will produce '
'additional log messages.',
))
config.declare('solvers', ConfigList(
domain=solver_config,
description='list of solvers to apply',
))
>>> config.display()
output: results.yml
verbose: 0
solvers: []
>>> print(config.generate_yaml_template())
output: results.yml # output results filename
verbose: 0 # output verbosity
solvers: [] # list of solvers to apply
It is important to note that both methods document the current state of
the configuration object. So, in the example above, since the solvers
list is empty, you will not get any information on the elements in the
list. Of course, if you add a value to the list, then the data will be
output:
>>> tmp = config()
>>> tmp.solvers.append({})
>>> tmp.display()
output: results.yml
verbose: 0
solvers:
-
iterlim: 100
lbfgs: true
linesearch: true
relative tolerance: 0.1
absolute tolerance: 0.2
>>> print(tmp.generate_yaml_template())
output: results.yml # output results filename
verbose: 0 # output verbosity
solvers: # list of solvers to apply
-
iterlim: 100 # iteration limit
lbfgs: true # use limited memory BFGS update
linesearch: true # use line search
relative tolerance: 0.1 # relative convergence tolerance
absolute tolerance: 0.2 # absolute convergence tolerance
Generating documentation
One of the most useful features of the Configuration system is the
ability to automatically generate documentation. To accomplish this, we
rely on a series of formatters derived from ConfigFormatter
that implement a visitor pattern for walking the hierarchy of
configuration containers (ConfigDict and ConfigList)
and documenting the members. As the ConfigFormatter was
designed to generate reference documentation, it behaves differently
from display() or
generate_yaml_template()):
For each configuration item, the doc field is output. If the
item has no doc, then the description field is used.
List containers have their domain documented and not their
current values.
The simplest interface for generating documentation is to call the
generate_documentation() method. This method
retrieves the specified formatter, instantiates it, and returns the
result from walking the configuration object. The documentation format
can be configured through optional arguments. The defaults generate
LaTeX documentation:
>>> print(config.generate_documentation())
\begin{description}[topsep=0pt,parsep=0.5em,itemsep=-0.4em]
\item[{output}]\hfill
\\output results filename
\item[{verbose}]\hfill
\\This sets the system verbosity. The default (0) only logs warnings and
errors. Larger integer values will produce additional log messages.
\item[{solvers}]\hfill
\\list of solvers to apply
\begin{description}[topsep=0pt,parsep=0.5em,itemsep=-0.4em]
\item[{iterlim}]\hfill
\\iteration limit
\item[{lbfgs}]\hfill
\\use limited memory BFGS update
\item[{linesearch}]\hfill
\\use line search
\item[{relative tolerance}]\hfill
\\relative convergence tolerance
\item[{absolute tolerance}]\hfill
\\absolute convergence tolerance
\end{description}
\end{description}
More useful is actually documenting the source code itself. To this
end, the Configuration system provides three decorators that append
documentation of the referenced ConfigDict (in
numpydoc format) for the most
common situations: