Monte Carlo Verification of Sensor Fusion

4.2.30 Monte Carlo Verification of Sensor Fusion

A deterministic sensor-fusion simulation can demonstrate that an estimator works for one particular set of initial conditions, sensor errors, model parameters, and timing assumptions. That is necessary, but it is not enough to establish that the estimator is robust.

Real navigation systems operate under uncertainty. Sensor biases vary from unit to unit, noise realizations change from run to run, mounting errors are never exactly zero, initialization is imperfect, measurements may be delayed or dropped, and the real spacecraft dynamics never match the estimator model perfectly.

Monte Carlo verification addresses this problem by repeating the complete sensor-fusion simulation many times while randomly varying the uncertain quantities according to justified statistical models.

The central idea is

\[ \boxed{ \text{One Simulation} \rightarrow \text{One Realization} } \]

whereas

\[ \boxed{ \text{Monte Carlo Simulation} \rightarrow \text{Statistical Distribution of Possible Outcomes} } \]

The objective is therefore not simply to ask

“Did the estimator work in this run?”

but rather

“How reliably does the estimator work across the full expected uncertainty environment?”

For spacecraft sensor fusion, Monte Carlo analysis is particularly important because the navigation solution is influenced simultaneously by sensor noise, bias, calibration uncertainty, dynamic-model error, timing uncertainty, initialization error, measurement availability, geometry, estimator tuning, and numerical behavior.

A useful high-level Monte Carlo architecture is

\[ \boxed{ \text{Nominal Scenario} + \text{Randomized Uncertainties} \rightarrow \text{Many Sensor-Fusion Runs} \rightarrow \text{Statistical Performance Assessment} } \]

4.2.30.1 Why Monte Carlo Verification Is Required

Suppose a navigation estimator is tested once using

\[ \hat{\mathbf x}_0 = \mathbf x_{true,0}, \]

nominal sensor noise, perfect timing, no measurement dropout, and nominal physical parameters.

The result may look excellent.

However, this does not demonstrate robustness because the estimator may behave very differently when

\[ \hat{\mathbf x}_0 \neq \mathbf x_{true,0}, \]

or when the actual sensor bias differs from its assumed nominal value.

A practical estimator must therefore be evaluated over many realizations such as

\[ \mathbf x_0^{(i)}, \qquad \mathbf b^{(i)}, \qquad \mathbf R^{(i)}, \qquad \mathbf Q^{(i)}, \qquad \tau^{(i)}, \qquad \boldsymbol{\epsilon}^{(i)}, \]

for Monte Carlo run i.

The result of one run describes behavior under one realization.

The distribution across all runs describes robustness.

Thus,

\[ \boxed{ \text{Nominal Accuracy} \neq \text{Robustness} } \]

4.2.30.2 Monte Carlo Run Definition

Let the uncertain parameter vector for run i be

\[ \boldsymbol{\theta}^{(i)} = \begin{bmatrix} \delta\mathbf x_0 & \mathbf b_0 & \boldsymbol{\eta}_{sensor} & \boldsymbol{\epsilon}_{misalign} & \boldsymbol{\delta}_{model} & \tau & \cdots \end{bmatrix}^{T}. \]

Each Monte Carlo realization draws

\[ \boldsymbol{\theta}^{(i)} \sim p(\boldsymbol{\theta}), \]

where p(\boldsymbol{\theta}) represents the assumed uncertainty distribution.

The complete simulation is then executed:

\[ \boxed{ \boldsymbol{\theta}^{(i)} \rightarrow \text{Truth Dynamics} \rightarrow \text{Sensors} \rightarrow \text{Estimator} \rightarrow \text{Navigation Performance} } \]

This is repeated for

\[ i=1,2,\ldots,N_{MC}. \]

The important point is that the estimator algorithm itself should normally remain unchanged between runs. The uncertainties change; the estimator is being tested against them.

Engineering interpretation: The run definition is the contract for repeatability: every randomized quantity should be traceable to a distribution, seed, and physical interpretation so that the same realization can be regenerated later.

4.2.30.3 Sensor Bias Variation

Sensor bias is one of the most important quantities to randomize.

For a gyroscope,

\[ \boldsymbol{\omega}_m = \boldsymbol{\omega}_{true} + \mathbf b_g + \mathbf n_g. \]

The initial bias may be sampled as

\[ \mathbf b_{g,0}^{(i)} \sim \mathcal N \left( \bar{\mathbf b}_g, \mathbf P_{b_g} \right). \]

Similarly, accelerometer bias can be varied as

\[ \mathbf b_{a,0}^{(i)} \sim \mathcal N \left( \bar{\mathbf b}_a, \mathbf P_{b_a} \right). \]

Monte Carlo analysis then reveals whether the estimator can correctly identify, track, or tolerate realistic bias uncertainty.

If the estimator contains bias states, the verification should examine both navigation-state performance and bias-estimation performance.

For example,

\[ \mathbf e_{b_g} = \mathbf b_{g,true} - \hat{\mathbf b}_g. \]

Persistent bias-estimation error may indicate poor observability, incorrect process-noise tuning, insufficient excitation, or a mismatch between the assumed and actual bias dynamics.

4.2.30.4 Sensor Noise Variation

Sensor noise should not be represented by one fixed realization.

For each Monte Carlo run, a different noise sequence should normally be generated.

For a measurement

\[ \mathbf z_k = h(\mathbf x_k) + \mathbf v_k, \]

with

\[ \mathbf v_k \sim \mathcal N(\mathbf 0,\mathbf R), \]

different runs should use independent realizations

\[ \mathbf v_k^{(1)}, \mathbf v_k^{(2)}, \ldots, \mathbf v_k^{(N_{MC})}. \]

This is important because a filter that appears excellent under one favorable noise realization may behave differently under another.

Monte Carlo analysis therefore measures performance over the distribution implied by the sensor noise model.

4.2.30.5 Noise-Parameter Uncertainty

The actual sensor noise may itself be uncertain.

Suppose the estimator assumes

\[ R_{filter}, \]

while the true sensor behaves according to

\[ R_{true}. \]

A realistic Monte Carlo campaign may therefore vary

\[ R_{true}^{(i)} \]

while keeping

\[ R_{filter} \]

fixed at the nominal tuning value.

This evaluates robustness to imperfect sensor characterization.

The distinction is important:

\[ \boxed{ R_{true} \neq R_{filter} } \]

is often more realistic than assuming they are exactly identical.

Engineering interpretation: This separation between truth noise and filter-assumed noise is essential because a filter is normally tuned using imperfect sensor characterization; verification should therefore test both stochastic noise realizations and uncertainty in the assumed statistics.

4.2.30.6 Sensor Misalignment

Sensor mounting is never perfectly aligned with the assumed body frame.

A true sensor-to-body transformation may be

\[ \mathbf C_{S}^{B,true} \]

while the estimator assumes

\[ \mathbf C_{S}^{B,est}. \]

The difference can be represented by a small rotation

\[ \delta\boldsymbol{\theta}_{SB}. \]

For each Monte Carlo run,

\[ \delta\boldsymbol{\theta}_{SB}^{(i)} \sim p(\delta\boldsymbol{\theta}_{SB}). \]

The true sensor model then uses the perturbed mounting orientation.

This is especially important for gyroscopes, accelerometers, star trackers, cameras, Sun sensors, magnetometers, and LiDAR systems because small alignment errors can appear as state-estimation biases.

4.2.30.7 Scale-Factor and Calibration Errors

Sensor scale-factor uncertainty can also be randomized.

For a measurement axis,

\[ z_m = (1+s)z_{true} + b + n. \]

The scale factor s may vary across the Monte Carlo population:

\[ s^{(i)} \sim \mathcal N(0,\sigma_s^2). \]

Other calibration parameters may include

\[ \text{lever arm}, \quad \text{camera intrinsics}, \quad \text{boresight alignment}, \quad \text{timing offset}, \quad \text{temperature coefficient}. \]

These uncertainties are often small individually, but they can become significant in high-accuracy navigation.

4.2.30.8 Initialization Error

A filter should never be verified only from perfect initial conditions.

The true initial state is

\[ \mathbf x_0, \]

while the estimator begins with

\[ \hat{\mathbf x}_0 = \mathbf x_0 + \delta\mathbf x_0. \]

Monte Carlo runs can sample

\[ \delta\mathbf x_0^{(i)} \sim \mathcal N (\mathbf 0,\mathbf P_{init}). \]

The uncertainty may involve

\[ \delta \mathbf r_0, \quad \delta \mathbf v_0, \quad \delta \boldsymbol{\theta}_0, \quad \delta \mathbf b_{g,0}, \quad \delta \mathbf b_{a,0}. \]

This allows verification of convergence behavior rather than merely steady-state accuracy.

4.2.30.9 Initial Covariance Uncertainty

The estimator's initial covariance

\[ \mathbf P_0 \]

also influences early behavior.

Monte Carlo verification should examine whether the assumed covariance realistically represents the actual spread of initialization errors.

Suppose across runs the actual initial estimation error covariance is approximately

\[ E[ (\mathbf x_0-\hat{\mathbf x}_0) (\mathbf x_0-\hat{\mathbf x}_0)^T ]. \]

This should be consistent with the selected

\[ \mathbf P_0. \]

If \mathbf P_0 is too small, the estimator becomes initially overconfident.

If it is excessively large, correction gains may become unnecessarily aggressive.

Engineering interpretation: The practical test is not only whether the filter converges, but whether the early correction behavior and reported confidence are compatible with the actual initialization-error population.

4.2.30.10 Timing Uncertainty

Measurement timing should also be randomized where appropriate.

Suppose a measurement generated at

\[ t_m \]

arrives at

\[ t_m+\tau. \]

The latency may vary:

\[ \tau^{(i)} \sim p(\tau). \]

Additional timing uncertainties can include

\[ \text{clock offset}, \quad \text{jitter}, \quad \text{timestamp error}, \quad \text{processing latency}, \quad \text{communication delay}. \]

A robust sensor-fusion architecture should remain consistent when these effects are within their expected ranges.

4.2.30.11 Measurement Dropout

Sensors may become temporarily unavailable.

Monte Carlo scenarios should therefore include dropout.

For sensor j,

\[ valid_j(k) = 0 \]

during selected intervals.

Dropout can be modeled using

• fixed-duration outages,
• randomly occurring outages,
• geometry-dependent availability,
• field-of-view loss,
• communication loss,
• or health-state transitions.

The estimator should continue propagating while measurement information is unavailable.

Uncertainty should generally grow during the outage.

When measurements return, reacquisition should occur without unacceptable transients.

4.2.30.12 Random Dropout Models

A simple stochastic dropout model may use probability

\[ P(dropout)=p_d. \]

At each measurement opportunity,

\[ d_k \sim \operatorname{Bernoulli}(p_d). \]

Then

\[ d_k=1 \]

may indicate that the measurement is unavailable.

More realistic models may use burst outages rather than independent dropouts because real communication or visibility losses often persist for several samples.

Engineering interpretation: Independent Bernoulli loss is convenient for basic testing, while burst or state-dependent dropout models are usually better for representing visibility loss, communication interruptions, and sustained sensor outages.

4.2.30.13 Dynamic-Model Uncertainty

The truth dynamics and estimator dynamics should not be identical.

Let

\[ \dot{\mathbf x}_{true} = f_{true} ( \mathbf x, \mathbf u, \mathbf p_{true} ), \]

while the estimator uses

\[ \dot{\hat{\mathbf x}} = f_{est} ( \hat{\mathbf x}, \mathbf u, \mathbf p_{nom} ). \]

Monte Carlo runs may vary

\[ \mathbf p_{true}^{(i)} = \mathbf p_{nom} + \delta\mathbf p^{(i)}. \]

Examples include

\[ \text{mass}, \quad \text{inertia}, \quad \text{drag parameters}, \quad \text{disturbance coefficients}, \quad \text{actuator characteristics}, \quad \text{environmental parameters}. \]

This tests whether process-noise modeling and measurement correction can compensate for realistic model mismatch.

4.2.30.14 Actuator Uncertainty

When sensor fusion is tested in closed-loop GNC, actuator uncertainty can affect navigation indirectly because control changes the trajectory.

For example,

\[ \mathbf u_{actual} = \mathbf u_{cmd} + \delta\mathbf u. \]

Possible uncertainties include

\[ \text{magnitude error}, \quad \text{misalignment}, \quad \text{delay}, \quad \text{deadband}, \quad \text{saturation}, \quad \text{minimum impulse behavior}. \]

These need not be estimator states, but they influence the truth trajectory and therefore the measurements available to the estimator.

4.2.30.15 Correlated Uncertainties

Not all uncertain quantities are independent.

For example, calibration errors may share common temperature dependence, or multiple axes may have correlated noise.

Instead of independently sampling

\[ x_i \sim \mathcal N(0,\sigma_i^2), \]

a correlated vector can be sampled as

\[ \mathbf e \sim \mathcal N ( \mathbf 0, \mathbf \Sigma ). \]

A covariance matrix containing off-diagonal terms then represents correlation.

Ignoring known correlations can produce an unrealistic Monte Carlo environment.

Engineering interpretation: Correlation matters because several individually reasonable one-dimensional uncertainty models can become physically unrealistic when combined independently. The sampled joint distribution should preserve known common-mode effects.

4.2.30.16 Choice of Probability Distributions

Gaussian uncertainty is common but not always appropriate.

Possible distributions include

\[ \mathcal N(\mu,\sigma^2), \]

uniform distributions,

\[ x \sim \mathcal U(a,b), \]

bounded distributions, discrete states, Bernoulli events, or empirically derived distributions.

The chosen distribution should represent the physical uncertainty source.

For example, manufacturing tolerance may be naturally bounded, whereas electronic noise may be approximately Gaussian.

4.2.30.17 Gaussian 3\sigma Interpretation

For a Gaussian scalar variable,

\[ x \sim \mathcal N(\mu,\sigma^2), \]

approximately 99.7\% of realizations lie within

\[ \mu \pm 3\sigma. \]

This is often used to define simulation uncertainty ranges.

However,

\[ \boxed{ 3\sigma \neq \text{physical hard bound} } \]

for a true Gaussian variable.

Monte Carlo assumptions should therefore distinguish between statistical standard deviation and absolute engineering limits.

4.2.30.18 Bounded Engineering Uncertainty

Some parameters have known engineering bounds.

For example,

\[ \delta\theta \in [-\theta_{\max},\theta_{\max}]. \]

A uniform model may then be more appropriate:

\[ \delta\theta \sim \mathcal U (-\theta_{\max},\theta_{\max}). \]

This is particularly useful when tolerance data are specified as allowable ranges rather than statistical standard deviations.

4.2.30.19 Monte Carlo Sample Size

The number of runs

\[ N_{MC} \]

must be large enough to estimate the desired performance statistics.

There is no universally sufficient number.

The required number depends on

\[ \text{dimensionality}, \quad \text{tail probability of interest}, \quad \text{metric variance}, \quad \text{required confidence}, \quad \text{computational cost}. \]

A few tens of runs may reveal obvious sensitivity.

Hundreds or thousands may be required for stable statistical estimates.

Very rare failure probability estimation may require substantially more sophisticated methods than ordinary Monte Carlo.

Engineering interpretation: Sample size should therefore be selected from the statistical claim being made rather than from a fixed rule of thumb. Estimating a mean accurately is much easier than demonstrating a very small failure probability.

4.2.30.20 Random Seed Management

Monte Carlo simulations should use controlled random seeds.

A useful configuration records

\[ seed_i \]

for every run.

Then a failed case can be reproduced exactly.

This is essential because an unexplained failure is difficult to debug if its random realization cannot be recreated.

Thus,

\[ \boxed{ \text{Monte Carlo Randomness} + \text{Recorded Seeds} = \text{Reproducible Verification} } \]

4.2.30.21 Navigation Error Metrics

For run i, define estimation error

\[ \mathbf e_k^{(i)} = \mathbf x_{true,k}^{(i)} - \hat{\mathbf x}_k^{(i)}. \]

Important components may include

\[ \mathbf e_r, \quad \mathbf e_v, \quad \mathbf e_\theta, \quad \mathbf e_\omega, \quad \mathbf e_b. \]

These errors should normally be evaluated over time and across runs.

The Monte Carlo campaign should not reduce all performance to one scalar too early.

Engineering interpretation: Keeping vector, component-wise, norm, peak, and time-history errors available prevents an apparently good aggregate score from hiding a directional or transient weakness.

4.2.30.22 Position Error

For translational navigation,

\[ \mathbf e_r = \mathbf r_{true} - \hat{\mathbf r}. \]

Useful metrics include component errors

\[ e_x, \quad e_y, \quad e_z, \]

and norm

\[ \boxed{ e_r = \|\mathbf e_r\|. } \]

The position error distribution can be examined at specific mission epochs or over the entire trajectory.

4.2.30.23 Velocity Error

Similarly,

\[ \mathbf e_v = \mathbf v_{true} - \hat{\mathbf v}, \]

with magnitude

\[ \boxed{ e_v = \|\mathbf e_v\|. } \]

Velocity error can be especially important in relative-navigation and proximity-operation applications because small velocity errors can integrate into substantial position deviations over time.

4.2.30.24 Attitude Error

Quaternion subtraction should not normally be used as the physical attitude-error metric.

Instead, define an attitude-error quaternion such as

\[ \delta q = q_{true} \otimes \hat q^{-1}. \]

The corresponding angular error can then be obtained from the quaternion.

For small errors,

\[ \delta\boldsymbol{\theta} \approx 2\delta\mathbf q_v. \]

A Monte Carlo campaign can then evaluate

\[ \|\delta\boldsymbol{\theta}\| \]

rather than individual quaternion-component differences.

4.2.30.25 Root-Mean-Square Error

For scalar error samples,

\[ RMSE = \sqrt{ \frac{1}{N} \sum_{k=1}^{N} e_k^2 }. \]

For each Monte Carlo run, this produces

\[ RMSE^{(i)}. \]

The distribution

\[ \{ RMSE^{(1)}, RMSE^{(2)}, \ldots, RMSE^{(N_{MC})} \} \]

can then be summarized using mean, median, percentile, standard deviation, and worst observed values.

4.2.30.26 Maximum Error

RMSE can hide short-duration peaks.

Therefore also evaluate

\[ e_{\max}^{(i)} = \max_k |e_k^{(i)}|. \]

Maximum error is particularly useful when navigation requirements specify hard transient limits.

A filter may have a low RMSE but still violate a critical maximum-error requirement during initialization or sensor transition.

4.2.30.27 Convergence Time

Define a convergence condition, for example

\[ \|\mathbf e(t)\| < e_{tol} \]

and

\[ \sigma(t) < \sigma_{tol} \]

for a required dwell interval.

The convergence time for run i is

\[ t_{conv}^{(i)}. \]

Monte Carlo analysis can then determine how initialization uncertainty and sensor conditions influence convergence.

Engineering interpretation: A dwell requirement prevents a momentary crossing of the tolerance threshold from being misclassified as convergence. This is especially important when noisy measurements cause the error to oscillate around the limit.

4.2.30.28 Navigation Availability

A useful operational metric is the fraction of time navigation remains valid.

For run i,

\[ A^{(i)} = \frac{T_{NavValid}^{(i)}}{T_{total}}. \]

Across all runs, this gives the distribution of navigation availability.

This captures effects that pure state-error metrics can miss, such as estimator invalidation, prolonged dropout, or health-monitor transitions.

4.2.30.29 Innovation Statistics

For sensor j,

\[ \boldsymbol{\nu}_{j,k} = \mathbf z_{j,k} - h_j(\hat{\mathbf x}_k^-). \]

Across Monte Carlo runs, innovations should be analyzed statistically.

Useful checks include

\[ E[\boldsymbol{\nu}], \]

innovation variance,

\[ Cov(\boldsymbol{\nu}), \]

autocorrelation, rejection frequency, and Normalised Innovation Squared.

A biased innovation distribution often reveals model or calibration mismatch.

Engineering interpretation: Innovation analysis is particularly useful because it checks the internal consistency of the estimator before a large state error necessarily becomes visible. Persistent structure in the residual often reveals a modelling problem early.

4.2.30.30 Innovation Covariance

The predicted innovation covariance is

\[ \boxed{ \mathbf S_k = \mathbf H_k \mathbf P_k^- \mathbf H_k^T + \mathbf R_k. } \]

The actual innovation dispersion across Monte Carlo runs should be broadly consistent with this prediction when the filter assumptions are valid.

This provides a powerful internal consistency check.

4.2.30.31 Normalised Innovation Squared

The NIS is

\[ \boxed{ \epsilon_{\nu,k} = \boldsymbol{\nu}_k^T \mathbf S_k^{-1} \boldsymbol{\nu}_k. } \]

If the measurement model is correct and the assumptions are satisfied, NIS follows a chi-square distribution with degrees of freedom equal to the measurement dimension:

\[ \epsilon_{\nu,k} \sim \chi^2_m. \]

This enables statistically meaningful consistency testing.

4.2.30.32 Why NIS Is Important

Small state-estimation error alone does not prove that the filter is statistically consistent.

Suppose the actual error is modest, but the estimator covariance is unrealistically tiny.

The filter is then overconfident.

NIS provides a way to compare innovation magnitude with the uncertainty the filter itself predicts.

Therefore,

\[ \boxed{ \text{Accuracy} \neq \text{Consistency} } \]

and both should be verified.

4.2.30.33 NIS Confidence Bounds

For significance level \alpha,

\[ \chi^2_{\alpha/2,m} \le \epsilon_{\nu} \le \chi^2_{1-\alpha/2,m} \]

provides a statistical interval for individual tests.

Across many Monte Carlo realizations and epochs, the fraction of values outside the expected interval can be compared with the theoretical significance level.

Persistent high NIS suggests underestimated uncertainty, model mismatch, outliers, or wrong measurement models.

Persistent very low NIS can indicate overly conservative uncertainty.

Engineering interpretation: The bounds should be interpreted statistically over an ensemble, not as a rule that every single sample must remain inside the interval. Occasional excursions are expected even for a correctly tuned filter.

4.2.30.34 Normalised Estimation Error Squared

When truth is available in simulation, NEES can be calculated:

\[ \boxed{ \epsilon_x = \mathbf e^T \mathbf P^{-1} \mathbf e. } \]

For a state dimension n,

\[ \epsilon_x \sim \chi^2_n \]

under ideal consistency assumptions.

NEES compares actual state-estimation error against the covariance claimed by the filter.

4.2.30.35 Why NEES Is Powerful

Suppose

\[ \|\mathbf e\| \]

is small.

That might appear satisfactory.

But if

\[ \mathbf P \]

is much smaller still, then the estimator is statistically overconfident.

Conversely, a large covariance paired with very small actual error may indicate excessive conservatism.

NEES therefore tests whether

\[ \boxed{ \text{Actual Error} \quad\text{matches}\quad \text{Predicted Uncertainty} } \]

rather than examining either quantity alone.

4.2.30.36 NIS Versus NEES

NIS uses only quantities normally available onboard:

\[ \boldsymbol{\nu}, \quad \mathbf S. \]

NEES requires truth:

\[ \mathbf x_{true}. \]

Therefore,

\[ \boxed{ \text{NIS} \rightarrow \text{measurement-space consistency} } \]

while

\[ \boxed{ \text{NEES} \rightarrow \text{state-space consistency}. } \]

NIS can be useful even with real flight data.

NEES is primarily a simulation or truth-referenced verification metric.

4.2.30.37 Averaged NIS

For N statistically independent samples,

\[ \bar{\epsilon}_{\nu} = \frac{1}{N} \sum_{i=1}^{N} \epsilon_{\nu}^{(i)}. \]

The average provides a more stable assessment of consistency than one instantaneous value.

The same concept can be applied across Monte Carlo runs, time samples, or both.

4.2.30.38 Averaged NEES

Similarly,

\[ \bar{\epsilon}_{x} = \frac{1}{N} \sum_{i=1}^{N} \epsilon_x^{(i)}. \]

The expected value is related to the state dimension when the estimator is consistent.

Systematic deviation indicates that the covariance model may not accurately represent actual estimation error.

4.2.30.39 Covariance Consistency

Another intuitive check compares state error against covariance envelopes.

For state x_j,

\[ \sigma_j = \sqrt{P_{jj}}. \]

Plot

\[ e_j = x_{true,j} - \hat x_j \]

with

\[ \pm 1\sigma_j, \qquad \pm 2\sigma_j, \qquad \pm 3\sigma_j. \]

Across many Monte Carlo runs, the empirical fraction of errors falling within these intervals can be compared with expected Gaussian coverage.

This produces a highly intuitive visualization of filter consistency.

Engineering interpretation: These plots connect actual estimation error with the uncertainty reported by the filter, making overconfidence and excessive conservatism visible in a form that is easy to interpret alongside NIS and NEES.

4.2.30.40 Monte Carlo Error Envelopes

At time t, suppose the position error from each run is

\[ e_r^{(1)}(t), \ldots, e_r^{(N)}(t). \]

Monte Carlo envelopes can show

\[ P_{5}(t), \quad P_{50}(t), \quad P_{95}(t), \]

where P_p denotes the p-th percentile.

These percentile envelopes reveal both typical and tail behavior.

They are often more informative than plotting hundreds of individual trajectories.

4.2.30.41 Mean Error

The Monte Carlo mean error is

\[ \bar{\mathbf e}(t) = \frac{1}{N_{MC}} \sum_{i=1}^{N_{MC}} \mathbf e^{(i)}(t). \]

Ideally,

\[ \bar{\mathbf e} \approx 0 \]

for an unbiased estimator.

Persistent non-zero mean can indicate

\[ \text{bias}, \quad \text{incorrect calibration}, \quad \text{model mismatch}, \quad \text{initialization asymmetry}, \quad \text{timing error}. \]

4.2.30.42 Empirical Error Covariance

Across runs,

\[ \mathbf P_{emp}(t) = \frac{1}{N_{MC}-1} \sum_{i=1}^{N_{MC}} \left( \mathbf e^{(i)}-\bar{\mathbf e} \right) \left( \mathbf e^{(i)}-\bar{\mathbf e} \right)^T. \]

This empirical covariance can be compared with the estimator's predicted covariance.

If the estimator consistently reports significantly smaller uncertainty than observed,

\[ \mathbf P_{filter} < \mathbf P_{emp}, \]

the filter may be overconfident.

4.2.30.43 Success Criteria

Every Monte Carlo campaign should define success before the simulations are run.

Success may depend on conditions such as

\[ e_r < e_{r,\max}, \]

\[ e_v < e_{v,\max}, \]

\[ e_\theta < e_{\theta,\max}, \]

\[ P_{ii} < P_{ii,\max}, \]

\[ NavValid = 1, \]

and absence of estimator divergence.

A run should not be judged subjectively after the result is seen.

Engineering interpretation: Success criteria should be fixed before examining the Monte Carlo results so that pass/fail classification remains objective and traceable to navigation or system requirements.

4.2.30.44 Run-Level Success

Define a binary outcome

\[ s_i = \begin{cases} 1,&\text{run meets all required criteria},\\ 0,&\text{otherwise}. \end{cases} \]

Then the observed success fraction is

\[ \boxed{ \hat p_s = \frac{1}{N_{MC}} \sum_{i=1}^{N_{MC}} s_i. } \]

This is one of the simplest mission-level Monte Carlo metrics.

However, the sample size and confidence interval must also be reported.

4.2.30.45 Confidence Intervals

An observed success rate from finite Monte Carlo simulations is only an estimate of the underlying probability.

For example, if

\[ 97 \]

of

\[ 100 \]

runs succeed, the underlying success probability is not proven to be exactly 97\%.

A statistical confidence interval should therefore accompany the estimate.

Thus,

\[ \boxed{ \text{Observed Success Fraction} \neq \text{Exact True Success Probability} } \]

Engineering interpretation: Reporting only the observed percentage can overstate the strength of the evidence. The interval communicates how much uncertainty remains because only a finite number of realizations were simulated.

4.2.30.46 Confidence Intervals for Continuous Metrics

For metrics such as RMSE or convergence time, uncertainty in the estimated mean can also be reported.

For sample mean

\[ \bar x \]

and sample standard deviation

\[ s, \]

the standard error is approximately

\[ SE = \frac{s}{\sqrt{N}}. \]

Confidence intervals then quantify how precisely the Monte Carlo population mean has been estimated.

4.2.30.47 Percentiles

Navigation verification often benefits from percentile metrics such as

\[ P_{50}, \quad P_{90}, \quad P_{95}, \quad P_{99}. \]

For example,

\[ P_{95}(e_r) \]

means 95\% of observed runs had position error below that value.

Percentiles are useful because they do not assume that the performance metric itself is normally distributed.

4.2.30.48 Worst-Case Observed Run

The maximum observed metric

\[ e_{\max} = \max_i e^{(i)} \]

should be inspected.

However,

\[ \boxed{ \text{Worst Observed Monte Carlo Case} \neq \text{Guaranteed Physical Worst Case} } \]

because ordinary Monte Carlo sampling does not necessarily explore every extreme corner of the uncertainty space.

Worst-case analysis and Monte Carlo analysis serve different purposes.

4.2.30.49 Sensitivity Analysis

Monte Carlo results can also be used to understand which uncertainties dominate performance.

Suppose output metric

\[ J \]

depends on uncertain parameters

\[ \theta_1, \theta_2,\ldots,\theta_p. \]

Statistical correlation, regression, variance decomposition, or structured sweeps can identify which parameters most strongly influence

\[ J. \]

This helps prioritize calibration, sensor selection, estimator tuning, and verification effort.

Engineering interpretation: Sensitivity results are most useful when they connect a dominant output variation to a physically interpretable uncertainty source, because that identifies whether improvement should come from calibration, sensing, modelling, or filter tuning.

4.2.30.50 One-Factor-at-a-Time Studies

Before full Monte Carlo, it can be useful to vary one source at a time:

\[ \text{bias only}, \]

then

\[ \text{noise only}, \]

then

\[ \text{misalignment only}, \]

and so forth.

This provides physical understanding.

Full Monte Carlo then combines all uncertainties.

Thus a strong verification program often uses

\[ \boxed{ \text{Deterministic Sensitivity Studies} \rightarrow \text{Combined Monte Carlo} } \]

rather than jumping directly to random combinations.

4.2.30.51 Combined Uncertainty

The most realistic campaign eventually combines multiple effects:

\[ \boxed{ \begin{aligned} &\text{initial-state error} + \text{sensor bias} + \text{sensor noise}\\ &+ \text{misalignment} + \text{timing uncertainty} + \text{dropout}\\ &+ \text{model uncertainty} + \text{actuator uncertainty} + \text{environmental variation}. \end{aligned} } \]

This tests the estimator under conditions closer to actual operation.

Combined effects are important because individual uncertainties can interact nonlinearly.

4.2.30.52 Filter Tuning Under Monte Carlo

Monte Carlo should not be used merely to search blindly for Q and R.

However, it is useful for evaluating whether a tuning choice performs robustly across uncertainty.

For example, candidate process-noise settings

\[ Q_1, Q_2, Q_3 \]

can each be tested over the same random realizations.

This enables meaningful comparison because the environmental cases are held constant.

4.2.30.53 Common Random Numbers

When comparing two estimator variants, using identical random realizations can greatly improve interpretability.

Estimator A and estimator B should experience the same

\[ \mathbf x_0^{(i)}, \quad \mathbf b^{(i)}, \quad \mathbf n^{(i)}, \quad \tau^{(i)}. \]

Then performance differences are more directly attributable to the estimator architecture rather than random scenario variation.

This is sometimes called use of common random numbers.

Engineering interpretation: Using the same random realizations makes estimator comparisons paired rather than independent; this removes much of the run-to-run environmental variation from the comparison.

4.2.30.54 Monte Carlo for Sensor Selection

Monte Carlo can also compare sensor-suite architectures.

For example:

\[ \text{IMU + GNSS} \]

versus

\[ \text{IMU + GNSS + camera} \]

versus

\[ \text{IMU + GNSS + camera + LiDAR}. \]

Metrics may include

\[ \text{accuracy}, \quad \text{availability}, \quad \text{consistency}, \quad \text{dropout tolerance}, \quad \text{convergence}. \]

This turns sensor selection into a quantitative system-level trade study.

4.2.30.55 Monte Carlo for Multi-Rate Fusion

Sensor-rate uncertainties and timing combinations can also be varied.

Examples include

\[ f_{IMU}, \quad f_{GNSS}, \quad f_{camera}, \quad f_{LiDAR}, \]

plus jitter and latency.

The objective is to confirm that estimator behavior does not rely on one ideal synchronous timing relationship.

4.2.30.56 Monte Carlo for Measurement Gating

Innovation-gating thresholds affect both outlier rejection and valid-measurement acceptance.

Monte Carlo analysis can quantify

\[ P(\text{reject valid measurement}) \]

and

\[ P(\text{accept invalid measurement}). \]

A gate that is too tight may remove useful information.

A gate that is too loose may allow bad measurements to corrupt the estimate.

Thus gating thresholds should be verified statistically.

Engineering interpretation: This is a direct way to tune the trade between protection and availability: a strict gate protects the state from bad data but can also discard valid information needed for convergence and observability.

4.2.30.57 Monte Carlo with Outliers

Outlier events can be injected with randomized time, magnitude, direction, or duration.

For example,

\[ \mathbf z_{fault} = \mathbf z + \Delta\mathbf z_{outlier}. \]

The verification should examine

\[ \text{detection}, \quad \text{rejection}, \quad \text{state disturbance}, \quad \text{recovery time}, \quad \text{false alarms}. \]

This connects Monte Carlo analysis with robust fusion and FDIR validation.

4.2.30.58 Monte Carlo with Sensor Failure

A sensor may be completely removed during selected runs.

The estimator should transition from

\[ \text{Nominal Sensor Set} \]

to

\[ \text{Reduced Sensor Set}. \]

The resulting impact on

\[ \text{accuracy}, \quad \text{observability}, \quad \text{covariance}, \quad \text{navigation validity} \]

should be quantified.

This provides evidence for graceful degradation.

4.2.30.59 Monte Carlo During General RPO

For general rendezvous and proximity operations, Monte Carlo verification may vary

\[ \text{relative-state initialization}, \]

\[ \text{relative sensor noise}, \]

\[ \text{bearing uncertainty}, \]

\[ \text{range uncertainty}, \]

\[ \text{camera/LiDAR availability}, \]

\[ \text{timing}, \]

\[ \text{geometry}, \]

and model parameters.

The relative navigation state may be

\[ \mathbf x_{rel} = \begin{bmatrix} \mathbf r_{rel}\\ \mathbf v_{rel}\\ \mathbf q_{rel}\\ \boldsymbol{\omega}_{rel} \end{bmatrix}. \]

The general purpose of Monte Carlo analysis is to verify whether the estimator remains accurate, consistent, observable, and operationally valid as range, geometry, and measurement availability change.

This is a general RPO principle and does not depend on any particular mission implementation.

4.2.30.60 Phase-Dependent Performance

A spacecraft estimator may behave differently in different operating regimes.

Monte Carlo statistics can therefore be computed separately for

\[ \text{far-range}, \quad \text{intermediate-range}, \quad \text{close-range}, \quad \text{terminal phases}. \]

A filter that performs well overall may still have a weak phase.

Phase-dependent metrics expose these local performance limitations.

4.2.30.61 Covariance Growth During Sensor Loss

When a major measurement source disappears, covariance should usually grow.

For example,

\[ P_{rr}(t) \uparrow \]

during prolonged position-measurement loss.

Monte Carlo verification should check whether this growth is physically reasonable and whether navigation-health logic responds before uncertainty exceeds operational limits.

Engineering interpretation: Correct covariance growth is a feature, not a failure. It communicates the loss of information to downstream navigation-health logic and should reflect how quickly uncertainty actually increases without the missing sensor.

4.2.30.62 Reacquisition Performance

After dropout, the estimator may receive measurements again.

Important Monte Carlo metrics include

\[ t_{reacquire}, \]

\[ e_{peak,reacquire}, \]

\[ \text{number of rejected measurements}, \]

and whether convergence returns smoothly.

A filter that handles dropout well but reacts badly during reacquisition is not fully robust.

4.2.30.63 Numerical Robustness Across Runs

Monte Carlo verification should also detect numerical failures.

Record events such as

\[ NaN, \quad Inf, \quad P \not\succeq 0, \quad \text{Cholesky failure}, \quad \|q\|\neq1, \quad \text{matrix singularity}. \]

Even if these occur rarely, they are important because rare numerical failures may become unacceptable in long-duration or safety-critical operation.

Engineering interpretation: Rare numerical failures should be treated seriously because a statistically good estimator that occasionally produces an invalid covariance or undefined state may still be unsuitable for autonomous operation.

4.2.30.64 Filter Divergence Detection

A Monte Carlo run should be flagged as diverged if defined criteria are violated.

Possible indicators include

\[ \|\mathbf e\| > e_{div}, \]

\[ P_{ii} > P_{div}, \]

persistent excessive NIS,

invalid covariance,

or invalid numerical state.

The divergence criterion should be explicit and consistent across all runs.

4.2.30.65 Run Classification

Each run can be classified as

\[ \boxed{ \text{Nominal} \quad \text{Degraded} \quad \text{Failed} } \]

rather than only pass/fail.

For example:

Nominal — all accuracy, consistency, and availability requirements satisfied.

Degraded — navigation remains usable but with reduced performance or temporary limit violations.

Failed — estimator divergence, navigation invalidity, or critical requirement violation.

This provides more diagnostic information than one binary outcome.

4.2.30.66 Mission-Level Metrics

Sensor-fusion performance should ultimately be connected to mission-level consequences.

Possible metrics include

\[ \text{navigation availability}, \]

\[ \text{guidance error}, \]

\[ \text{control effort}, \]

\[ \text{pointing accuracy}, \]

\[ \text{relative-navigation accuracy}, \]

or whether downstream GNC requirements remain satisfied.

The objective is not merely to show that the covariance matrix behaves correctly, but that the navigation system supports the required operation.

Engineering interpretation: This connection is important because estimator accuracy is not an end in itself. The navigation solution is valuable only insofar as it supports the required guidance, control, pointing, or relative-navigation function.

4.2.30.67 Estimator Metrics and GNC Metrics

A strong Monte Carlo campaign distinguishes between

\[ \boxed{ \text{Estimator Metrics} } \]

and

\[ \boxed{ \text{System-Level GNC Metrics}. } \]

Estimator metrics may include

\[ RMSE, \quad NIS, \quad NEES, \quad P, \quad \text{convergence}. \]

System-level metrics may include

\[ \text{tracking error}, \quad \text{control effort}, \quad \text{mission-state success}, \quad \text{navigation validity}. \]

Both levels are needed.

4.2.30.68 Open-Loop Monte Carlo

Before closed-loop testing, Monte Carlo analysis can be performed using a prescribed truth trajectory.

\[ \boxed{ \text{Truth Trajectory} \rightarrow \text{Sensor Models} \rightarrow \text{Estimator} } \]

This isolates estimator behavior from controller interaction.

It is particularly useful during early tuning and model validation.

4.2.30.69 Closed-Loop Monte Carlo

The next level is

\[ \boxed{ \text{Plant} \rightarrow \text{Sensors} \rightarrow \text{Estimator} \rightarrow \text{Guidance} \rightarrow \text{Control} \rightarrow \text{Plant} } \]

with uncertainties randomized across runs.

This is more realistic because estimator error influences control, and control modifies future measurements.

Closed-loop Monte Carlo therefore verifies the complete coupled GNC system.

4.2.30.70 Parallel Execution

Monte Carlo simulations are naturally parallel because run i is usually independent of run j.

Thus

\[ Run_1, Run_2, \ldots, Run_N \]

can be distributed across multiple processor cores or compute nodes.

This can substantially reduce total verification time.

However, deterministic seed management and result indexing must be preserved.

Engineering interpretation: Parallel execution changes computational throughput, not the statistical definition of the campaign. Run identity, seeds, configuration, and result ordering should therefore remain deterministic even when jobs finish out of order.

4.2.30.71 Monte Carlo Data Storage

Saving every signal from every run can become expensive.

A good strategy is to retain key time histories such as

\[ \mathbf e(t), \quad P(t), \quad NIS(t), \quad NavValid(t), \]

while also storing compact run-level metrics.

For failed or unusual cases, full logging can be retained for diagnosis.

This balances traceability with storage efficiency.

4.2.30.72 Run Metadata

Every run should record configuration information such as

\[ runID, \quad seed, \quad parameter\ set, \quad software\ version, \quad scenario, \quad result. \]

Without this metadata, a statistical result may be difficult to reproduce or audit later.

4.2.30.73 Failed-Run Investigation

Monte Carlo verification should never stop at

“3 runs failed.”

Each failed run should be reproducible and investigated.

Questions include:

• Which uncertainty realization triggered the failure?
• Did the estimator become inconsistent before divergence?
• Was a measurement wrongly accepted or rejected?
• Did covariance become unrealistic?
• Was the issue observability, timing, tuning, geometry, or numerical failure?

Failure investigation often provides more engineering value than the overall success percentage.

Engineering interpretation: A failed run is often more informative than many nominal runs because it exposes the combination of uncertainty, geometry, timing, or numerical behavior that drives the system toward its limit.

4.2.30.74 Scatter Plots

Scatter plots can reveal relationships between uncertain inputs and output performance.

For example,

\[ b_{gyro,0} \quad \text{versus} \quad RMSE_\theta. \]

or

\[ \tau_{camera} \quad \text{versus} \quad e_{r,max}. \]

Visible trends can identify important sensitivity drivers.

4.2.30.75 Histograms

Histograms are useful for quantities such as

\[ RMSE, \quad t_{conv}, \quad e_{max}, \quad NIS, \quad NEES. \]

They reveal whether performance is approximately Gaussian, skewed, multimodal, or heavy-tailed.

This matters because mean and standard deviation alone may hide important distribution structure.

4.2.30.76 Empirical Cumulative Distribution Function

The empirical CDF is particularly useful.

For performance metric J,

\[ F_J(x) = P(J\le x). \]

From it one can directly read quantities such as

\[ P(J

This provides an intuitive connection between Monte Carlo statistics and engineering requirements.

4.2.30.77 Time-History Confidence Envelopes

For a state-error time history, envelopes can be plotted across runs.

For example,

\[ e_{50}(t), \quad e_{95}(t), \quad e_{99}(t). \]

This reveals when uncertainty becomes largest during the trajectory.

Such envelopes are especially useful for mission-phase analysis.

4.2.30.78 Requirement Verification

Each requirement should map to a measurable Monte Carlo quantity.

For example:

\[ \|\mathbf e_r\|

\[ \|\mathbf e_v\|

\[ \|\delta\boldsymbol{\theta}\|<\theta_{req}, \]

\[ NavValid=1. \]

Verification then becomes a traceable statement of whether the run population satisfies these requirements.

4.2.30.79 Statistical Requirement Interpretation

If a requirement is probabilistic, such as

\[ P(e_r0.99, \]

then the Monte Carlo sample size must support meaningful estimation of that probability.

The number of runs and confidence interval therefore become part of the verification argument.

A very high claimed probability cannot be convincingly demonstrated with only a handful of simulations.

Engineering interpretation: The tighter the claimed probability, the stronger the evidence required. Sample count, observed failures, and the chosen confidence method therefore form part of the requirement-verification result.

4.2.30.80 Monte Carlo Is Not a Substitute for Model Validation

Running thousands of simulations does not make an incorrect model correct.

If the sensor-noise distribution, bias dynamics, calibration uncertainty, or truth dynamics are unrealistic, the Monte Carlo results will also be unrealistic.

Therefore,

\[ \boxed{ \text{Monte Carlo Quality} \le \text{Model Quality} } \]

in the practical sense that statistical verification cannot compensate for poor underlying assumptions.

Engineering interpretation: Statistical repetition only explores the model that was supplied. If the physical assumptions are wrong, additional runs simply characterize the wrong system more precisely.

4.2.30.81 Monte Carlo Is Not the Same as Worst-Case Analysis

Monte Carlo estimates probabilistic behavior based on sampled distributions.

Worst-case analysis searches for extreme allowable combinations.

These answer different questions.

Monte Carlo asks:

\[ \text{What happens with high probability?} \]

Worst-case analysis asks:

\[ \text{What is the most adverse allowable condition?} \]

A strong verification program may use both.

4.2.30.82 Corner Cases

Specific corner cases should still be tested deliberately even when their random probability is low.

Examples include

\[ \text{maximum bias}, \quad \text{maximum delay}, \quad \text{minimum visibility}, \quad \text{worst initialization}, \quad \text{simultaneous sensor loss}. \]

These should not be left entirely to chance sampling.

4.2.30.83 Statistical Independence

When combining results, care should be taken regarding independence.

Multiple time samples from one run are not necessarily statistically independent.

Similarly, correlated measurement sequences should not be treated as independent observations when constructing confidence arguments.

This matters especially for formal statistical interpretation of NIS, NEES, and confidence intervals.

Engineering interpretation: This matters when constructing confidence bounds: treating many correlated time samples from one realization as if they were independent can make the evidence appear much stronger than it actually is.

4.2.30.84 Measurement Correlation

If measurement noise is correlated,

\[ E[\mathbf v_i\mathbf v_j^T] \neq0, \]

the Monte Carlo sensor model should reproduce that correlation where relevant.

Otherwise the simulated environment may be easier than the real one.

This is particularly important when several measurements share common processing or common physical error sources.

4.2.30.85 Process-Noise Realizations

Truth disturbances can also be randomized.

For example,

\[ \mathbf w_k \sim \mathcal N(\mathbf 0,Q_{true}). \]

The estimator may use

\[ Q_{filter}. \]

Comparing performance for different

\[ Q_{true} \]

relative to

\[ Q_{filter} \]

helps evaluate robustness to process-model uncertainty.

4.2.30.86 Confidence Calibration

A well-calibrated estimator should satisfy the intuitive condition

\[ \boxed{ \text{Reported Uncertainty} \approx \text{Observed Error Statistics}. } \]

If actual errors repeatedly exceed predicted 3\sigma bounds, the filter is overconfident.

If actual errors are consistently much smaller than predicted uncertainty, the estimator may be overly conservative.

Monte Carlo simulation is one of the best ways to examine this calibration.

Engineering interpretation: A calibrated filter does not merely produce small error; it reports uncertainty that has the correct statistical relationship to that error. This is why covariance validation is as important as state-error validation.

4.2.30.87 Conservative Filters

Conservative covariance is not automatically desirable.

Very large covariance can produce large Kalman gains, increased measurement sensitivity, or unnecessary degraded-mode declarations.

The objective is not

\[ \text{make }P\text{ as large as possible}, \]

but rather

\[ \boxed{ P \text{ should realistically represent estimation uncertainty}. } \]

4.2.30.88 Overconfident Filters

An overconfident filter may show

\[ P_{filter} \ll P_{empirical}. \]

This can cause

• excessive rejection of otherwise valid measurements,
• underestimated navigation risk,
• inappropriate downstream GNC decisions,
• or eventual divergence.

NIS and NEES statistics are particularly valuable for detecting this condition.

4.2.30.89 Mission-Phase Conditional Statistics

Instead of only reporting aggregate performance, statistics can be conditioned on phase:

\[ J_{far}, \quad J_{mid}, \quad J_{close}, \quad J_{terminal}. \]

This prevents one long, easy mission segment from dominating average metrics and hiding poorer performance in a short but critical phase.

4.2.30.90 Sensor-Availability Conditional Statistics

Similarly, compute performance for cases such as

\[ \text{all sensors available}, \]

\[ \text{GNSS unavailable}, \]

\[ \text{camera unavailable}, \]

\[ \text{LiDAR unavailable}. \]

This quantifies graceful degradation and redundancy value.

4.2.30.91 Configuration Management

Monte Carlo results depend on

\[ \text{software version}, \quad \text{estimator tuning}, \quad \text{sensor model}, \quad \text{uncertainty distributions}, \quad \text{scenario version}. \]

All of these should be recorded.

Otherwise statistical results cannot be reproduced reliably after the model changes.

Engineering interpretation: Versioning the complete statistical configuration is essential because a small change to a distribution, sensor model, threshold, or software revision changes the population being verified.

4.2.30.92 Automated Monte Carlo Pipeline

A useful automated workflow is

\[ \boxed{ \text{Generate Run Parameters} \rightarrow \text{Execute Simulation} \rightarrow \text{Extract Metrics} \rightarrow \text{Classify Run} \rightarrow \text{Aggregate Statistics} \rightarrow \text{Generate Report} } \]

Automation is important because manual handling becomes impractical for hundreds or thousands of simulations.

4.2.30.93 Pre-Run Validation

Before launching a large campaign, validate each generated parameter set.

For example,

\[ m>0, \]

\[ P_0\succeq0, \]

\[ R\succeq0, \]

\[ Q\succeq0, \]

and all sampled physical parameters should remain within allowed ranges.

This prevents invalid random draws from being incorrectly counted as estimator failures.

Engineering interpretation: Pre-run checks protect the integrity of the campaign by separating invalid test generation from genuine estimator failure. Otherwise, nonphysical samples can distort the observed failure rate.

4.2.30.94 Post-Run Validation

Each completed run should also pass basic integrity checks before inclusion in the statistics.

Examples include

\[ \text{simulation completed}, \]

\[ \text{no missing logs}, \]

\[ \text{time vector valid}, \]

\[ \text{metrics computable}, \]

\[ \text{no unintended numerical corruption}. \]

A simulation crash is itself important but should be classified correctly rather than silently mixed into normal metric distributions.

Engineering interpretation: Post-run checks serve the same purpose at the output side: they distinguish an estimator result from a simulation infrastructure problem and ensure that only interpretable runs enter the statistics.

4.2.30.95 Monte Carlo Verification Matrix

A useful verification matrix can group uncertainties by category:

\[ \boxed{ \begin{array}{c|c} \text{Category} & \text{Examples}\\ \hline \text{Initialization} & \mathbf r_0,\mathbf v_0,\mathbf q_0,\mathbf b_0\\ \text{Sensors} & \text{noise, bias, misalignment, scale factor}\\ \text{Timing} & \text{latency, jitter, clock offset}\\ \text{Availability} & \text{dropout, outage, FOV loss}\\ \text{Dynamics} & \text{mass, inertia, disturbance parameters}\\ \text{Estimator} & Q,R,P_0,\text{thresholds}\\ \text{Actuation} & \text{magnitude, delay, misalignment}\\ \end{array} } \]

This helps ensure that important uncertainty sources are not omitted.

4.2.30.96 Verification Progression

A disciplined sequence is

\[ \boxed{ \text{Deterministic Nominal} \rightarrow \text{Single-Uncertainty Sweeps} \rightarrow \text{Combined Monte Carlo} \rightarrow \text{Faulted Monte Carlo} \rightarrow \text{Closed-Loop Monte Carlo} } \]

Each stage answers a different question.

This progression makes failures easier to diagnose.

4.2.30.97 What Should Be Plotted

A strong sensor-fusion Monte Carlo report should normally include

• state-error envelopes,
• covariance envelopes,
• error versus 3\sigma,
• NIS distributions,
• NEES distributions,
• RMSE histograms,
• convergence-time distributions,
• maximum-error distributions,
• navigation-validity statistics,
• success/failure counts,
• percentile tables,
• and sensitivity plots.

These plots together provide far more information than one nominal trajectory.

4.2.30.98 Recommended Numerical Outputs

For every run, useful scalar outputs include

\[ RMSE_r, \quad RMSE_v, \quad RMSE_\theta, \]

\[ e_{r,max}, \quad e_{v,max}, \quad e_{\theta,max}, \]

\[ t_{conv}, \quad A_{nav}, \quad NIS_{mean}, \quad NEES_{mean}, \]

\[ N_{reject}, \quad N_{dropout}, \quad success. \]

These can be assembled into a Monte Carlo results table for statistical analysis.

4.2.30.99 Failure-Mode Statistics

The campaign should not only count failures but classify causes.

For example,

\[ N_{divergence}, \quad N_{timeout}, \quad N_{covariance}, \quad N_{observability}, \quad N_{numerical}, \quad N_{sensor}. \]

This reveals where the navigation system is fragile.

A 95\% overall success rate means little if all failures share one systematic and correctable cause.

4.2.30.100 Verification of Covariance Bounds

For each state component,

\[ e_i^{(j)}(t) \]

can be compared with

\[ 3\sqrt{P_{ii}^{(j)}(t)}. \]

Across all runs and times, compute the fraction satisfying

\[ |e_i| \le 3\sigma_i. \]

For a well-calibrated Gaussian system, this should broadly agree with expected coverage, subject to correlation and finite-sample effects.

4.2.30.101 Verification of Innovation Gates

For a selected chi-square gate,

\[ \gamma_\alpha, \]

measure the empirical rejection frequency for nominal measurements.

If nominal measurements are rejected far more often than expected, then

\[ R, \quad P, \quad H, \quad \text{timing}, \]

or the gate itself may be inconsistent.

Engineering interpretation: Comparing empirical rejection frequency against the designed chi-square probability checks the entire chain of covariance prediction, measurement noise modelling, residual calculation, timing, and threshold selection.

4.2.30.102 Monte Carlo for Adaptive Filters

If the estimator changes Q, R, sensor weighting, or mode online, Monte Carlo verification becomes even more important.

The adaptation law itself introduces another dynamic system.

The analysis should verify

\[ \text{adaptation stability}, \]

\[ \text{adaptation bounds}, \]

\[ \text{recovery behavior}, \]

and whether adaptation improves performance rather than merely hiding modeling errors.

4.2.30.103 Monte Carlo for Federated or Multi-Filter Architectures

For federated or distributed filters, Monte Carlo analysis should consider

\[ \text{local-filter consistency}, \]

\[ \text{cross-correlation}, \]

\[ \text{information sharing}, \]

\[ \text{fault isolation}, \]

and

\[ \text{master-filter performance}. \]

The complete architecture should be randomized and verified, not only each local estimator independently.

4.2.30.104 Monte Carlo for Non-Gaussian Filters

Particle filters or other non-Gaussian methods may require different consistency metrics.

Nevertheless, Monte Carlo verification remains fundamental.

One can still examine

\[ \text{state error}, \]

\[ \text{credible-interval coverage}, \]

\[ \text{effective sample size}, \]

\[ \text{degeneracy}, \]

\[ \text{convergence}, \]

and system-level navigation performance.

4.2.30.105 Monte Carlo and Observability

A filter may perform well in some geometric conditions and poorly in others because observability changes.

Randomizing geometry, motion, measurement availability, or initial conditions can expose these cases.

Poor Monte Carlo outcomes may therefore reflect

\[ \boxed{ \text{Estimator Weakness} \quad\text{or}\quad \text{Fundamental Observability Limitation}. } \]

These should not be confused.

Engineering interpretation: This distinction is critical because no amount of covariance tuning can recover information that the sensor geometry and motion do not make observable. Monte Carlo can reveal the condition, but the remedy may require different sensing or excitation.

4.2.30.106 Geometry-Dependent Performance

Camera, LiDAR, GNSS, and relative-navigation sensors often have geometry-dependent sensitivity.

The measurement Jacobian

\[ H \]

changes with geometry.

Monte Carlo analysis should therefore sample representative geometries rather than one convenient alignment.

This is especially important for range-only, bearing-only, and relative-pose estimation.

4.2.30.107 Long-Duration Monte Carlo

Some filters behave well over short runs but degrade over long duration because of

\[ \text{bias accumulation}, \quad \text{clock drift}, \quad \text{numerical roundoff}, \quad \text{slow covariance inconsistency}. \]

Long-duration Monte Carlo can expose these effects.

It is often computationally expensive but important for missions requiring prolonged autonomous navigation.

4.2.30.108 Rare-Event Limitations

Ordinary Monte Carlo is inefficient for extremely rare events.

If one wants to estimate probabilities such as

\[ 10^{-6}, \]

millions of ordinary runs may be required.

Methods such as importance sampling, subset simulation, or targeted stress testing may then be more appropriate.

For ordinary sensor-fusion verification, however, standard Monte Carlo remains extremely valuable for moderate-probability uncertainty assessment.

4.2.30.109 Verification Versus Validation

Monte Carlo can support both, but they are conceptually different.

Verification asks whether the estimator implementation behaves according to its mathematical and software design.

Validation asks whether the overall model represents the real navigation system sufficiently well for the intended use.

Thousands of runs can verify a model implementation statistically, but physical validation still requires realistic sensor characteristics and, eventually, representative hardware or measured data.

Engineering interpretation: A complete program therefore progresses from mathematical and software verification toward representative sensor data, processors, interfaces, and hardware so that the assumptions used in simulation are progressively confronted with reality.

4.2.30.110 Connection to SIL and HIL

Monte Carlo verification naturally prepares the system for

\[ \text{Software-in-the-Loop}, \]

\[ \text{Processor-in-the-Loop}, \]

and

\[ \text{Hardware-in-the-Loop}. \]

The same uncertainty cases can later be replayed against actual implementation code or hardware.

This creates continuity between model-level verification and implementation-level validation.

Engineering interpretation: The value of this continuity is that the same uncertainty cases, seeds, metrics, and acceptance criteria can be carried forward as the implementation becomes more flight representative.

4.2.30.111 Recommended Monte Carlo Workflow

A practical workflow is:

1. define the nominal scenario,
2. identify uncertainty sources,
3. assign physically justified distributions,
4. define correlations where required,
5. define deterministic success criteria,
6. define statistical metrics,
7. assign random seeds,
8. generate run parameters,
9. execute simulations,
10. log state, covariance, innovations, status, and GNC quantities,
11. reject invalid simulation runs only for clearly defined setup failures,
12. calculate run-level metrics,
13. evaluate NIS and NEES,
14. calculate percentiles and confidence intervals,
15. identify failures,
16. reproduce failed seeds,
17. perform sensitivity analysis,
18. verify requirements,
19. document the complete configuration.

This workflow makes Monte Carlo verification traceable rather than simply producing a large collection of plots.

4.2.30.112 Complete Monte Carlo Architecture

The overall architecture can be summarized as

\[ \boxed{ \begin{array}{c} \text{Define Uncertainty Models}\\ \downarrow\\ \text{Generate Random Realization}\\ \downarrow\\ \text{Configure Truth + Sensors + Estimator}\\ \downarrow\\ \text{Run Complete Sensor-Fusion Simulation}\\ \downarrow\\ \text{Compute State Errors}\\ \downarrow\\ \text{Compute RMSE / Maximum Error / Convergence}\\ \downarrow\\ \text{Compute NIS / NEES / Covariance Consistency}\\ \downarrow\\ \text{Check Navigation and Mission Criteria}\\ \downarrow\\ \text{Store Run Metrics}\\ \downarrow\\ \text{Repeat for }N_{MC}\text{ Runs}\\ \downarrow\\ \text{Aggregate Statistics}\\ \downarrow\\ \text{Confidence Intervals + Percentiles + Failure Analysis} \end{array} } \]

This is the core statistical verification loop.

4.2.30.113 Key Takeaways

Monte Carlo verification transforms sensor-fusion testing from one deterministic demonstration into a statistical assessment of robustness.

The most important uncertainty sources include

\[ \boxed{ \text{Sensor Bias} + \text{Noise} + \text{Misalignment} + \text{Initialization Error} + \text{Timing} + \text{Dropouts} + \text{Dynamic Uncertainty} } \]

together with calibration, actuator, environmental, and configuration uncertainties where relevant.

The estimator should be evaluated using actual state-estimation error, RMSE, peak error, convergence, innovation statistics, covariance consistency, NIS, NEES, navigation availability, failure classification, percentile metrics, and mission-level criteria.

NIS evaluates measurement-space consistency:

\[ \boxed{ \epsilon_\nu = \nu^TS^{-1}\nu } \]

while NEES evaluates state-space consistency:

\[ \boxed{ \epsilon_x = e^TP^{-1}e. } \]

Monte Carlo verification should include both accuracy and uncertainty calibration.

A filter that produces a small state error but unrealistic covariance is not fully verified.

A filter that reports conservative covariance but repeatedly violates operational navigation requirements is also not sufficient.

The central engineering principle is therefore

\[ \boxed{ \text{A robust navigation estimator must be accurate, consistent, available, numerically stable, and reliable across the expected uncertainty population.} } \]

Finally, Monte Carlo verification should be reproducible. Every run should retain its random seed, parameter realization, model version, estimator configuration, and result classification so that unusual or failed cases can be replayed and investigated.

This provides the natural transition to 4.2.31 SIL/HIL and Flight Validation, where the statistically verified estimator architecture is exercised using real implementation code, representative processors, interfaces, hardware, recorded sensor data, timing effects, and progressively more flight-like environments.

Next: 4.2.31 SIL/HIL and Flight Validation

Monte Carlo verification has now established statistical confidence in the estimator across uncertainty, noise, bias, timing variation, dropouts, model mismatch, numerical behavior, consistency metrics, and mission-level acceptance criteria.

The next section moves from model-level statistical verification toward increasingly flight-representative validation using software-in-the-loop, processor-in-the-loop, hardware-in-the-loop, sensor emulation, real interfaces, communication timing, recorded-data replay, and flight-test evidence.