The Python package called parmest facilitates model-based parameter
estimation along with characterization of uncertainty associated with
the estimates. For example, parmest can provide confidence regions
around the parameter estimates. Additionally, parameter vectors, each
with an attached probability estimate, can be used to build scenarios
for design optimization.
Background
The goal of parameter estimation is to estimate values for
a vector, \(\boldsymbol{\theta}\), to use in the functional form
\[\boldsymbol{y}_i = \boldsymbol{f}\left(\boldsymbol{x}_{i}, \boldsymbol{\theta}\right) +
\boldsymbol{\varepsilon}_i \quad \forall \; i \in \{1, \ldots, n\}\]
where \(\boldsymbol{y}_{i} \in \mathbb{R}^m\) are observations of the measured or output variables,
\(\boldsymbol{f}\) is the model function, \(\boldsymbol{x}_{i} \in \mathbb{R}^{q}\) are the decision
or input variables, \(\boldsymbol{\theta} \in \mathbb{R}^p\) are the model parameters,
\(\boldsymbol{\varepsilon}_{i} \in \mathbb{R}^m\) are measurement errors, and \(n\) is the number of
experiments.
The following least squares objective can be used to estimate parameter
values assuming Gaussian independent and identically distributed measurement
errors:
\[\min_{\boldsymbol{\theta}} \, g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta}) \;\;\]
where \(g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta})\) can be:
Sum of squared errors
\[g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta}) =
\sum_{i = 1}^{n} \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})
\right)^\text{T} \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right)\]
Weighted sum of squared errors
\[g(\boldsymbol{x}, \boldsymbol{y};\boldsymbol{\theta}) =
\frac{1}{2} \sum_{i = 1}^{n} \left(\boldsymbol{y}_{i} - \boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})
\right)^\text{T} \boldsymbol{\Sigma}_{\boldsymbol{y}}^{-1} \left(\boldsymbol{y}_{i} -
\boldsymbol{f}(\boldsymbol{x}_{i};\boldsymbol{\theta})\right)\]
where \(\boldsymbol{\Sigma}_{\boldsymbol{y}}\) is the measurement error covariance matrix containing the
standard deviation of the measurement errors of \(\boldsymbol{y}\). Custom objectives can also be defined
for parameter estimation.
In the applications of interest to us, the function \(g(\cdot)\) is
usually defined as an optimization problem with a large number of
(perhaps constrained) optimization variables, a subset of which are
fixed at values \(\boldsymbol{x}\) when the optimization is performed.
In other applications, the values of \(\boldsymbol{\theta}\) are fixed
parameter values, but for the problem formulation above, the values of
\(\boldsymbol{\theta}\) are the primary optimization variables. Note that in
general, the function \(g(\cdot)\) will have a large set of
parameters that are not included in \(\boldsymbol{\theta}\).