stochastic process🚧
stochastic process
專著《Introduction to Statistical Signal Processing》。
數字number：一個數字。 隨機變數random variable：一個浮動數字，擁有浮動範圍。（既非隨機、亦非變數） 分布distribution：浮動範圍。 數列sequence：一串數字。 隨機過程stochastic process：一串浮動數字，各自擁有浮動範圍。 聯合分布joint distribution：所有浮動範圍，整合成高維度分布。
大家應該聽過「實數」推廣成「複數」。 此處則是「數字」推廣成「分布」。 各種數學概念，必須從頭開始一一檢查是否仍然可行。 等於、四則運算、極限、微積分、收斂、遞迴關係、……。 而且也要一一檢查新的數學概念： 期望值、動差（平均數、變異數）、相干（共相關數）、……。 各種數學概念，都要重新翻譯，改寫成機率分布函數。 完全變成另外一種世界觀。 國情不同、語言不同、思路不同、價值不同。 中國人講倫理道德、孝順父母。 美國人講因果邏輯、獨立自主。 即便過著同樣生活，卻講著不同內容。 大概是這種感覺。
從「數字」的角度來看，隨機過程大致有兩個重點： 一、相等關係（常數函數）。 二、遞迴關係（遞迴函數）。 從「分布」的角度來看，這兩件事情融合在一起，無法分開談論。 如果硬要分開談論，整件事情形成下表。 內容混亂，看看就好，不必太往心裡去。
1. constant function -> identically distributed process 2. zero-order recurrence -> mutually independent process 3. first-order recurrence -> Markov process 4. time-invariant recurrence -> stationary process forms a constant function
一般來說，recurrence預設為time-invariant。 為了方便翻譯，此處權且將recurrence預設成time-variant。
stationary process
constant function / time-invariant function
in K-12 mathematics,
(1) constant = value remains unchanged
(2) constant/variable = value is known/unknown
in mathematics,
(1) constant function = output remains unchanged
by different input
(2) time-invariant function = output remains unchanged
by different time-input
= function remains unchanged
by different time
in stochastic process,
(1) stationary process ≈ non-changing
(constant & time-invariant)
(2) mutually independent process ≈ non-regressive
(acausal)
這兩個名詞無法直接翻譯。只是大概是這種感覺。
stationary process / mutually independent process
definition: stationary: independent between variables and time. mutually independent: independent between variable and variable. property: stationary => identically distributed mutually independent => pairwise independent
identically distributed process = constant distribution
X₁ = X₂ = X₃ = ...
stationary process = time-invariant joint distribution
X₁ = X₂ = X₃ = ...
(X₁, X₂) = (X₂, X₃) = (X₃, X₄) = ...
(X₁, X₂, X₃) = (X₂, X₃, X₄) = (X₃, X₄, X₅) = ...
:
pairwise independent process = constant marginal distribution
in 2-joint distribution
X₁ = (X₁, X₂ = x) for all x
X₁ = (X₁, X₃ = x) for all x
X₁ = (X₁, X₄ = x) for all x
: :
X₂ = (X₂, X₁ = x) for all x
X₂ = (X₂, X₃ = x) for all x
X₂ = (X₂, X₄ = x) for all x
: :
mutually independent process = constant marginal distribution
in joint distribution
X₁ = (X₁, X₂ = x₂) for all x₂
X₁ = (X₁, X₃ = x₃) for all x₃
X₁ = (X₁, X₄ = x₄) for all x₄
:
X₁ = (X₁, X₂ = x₂, X₃ = x₃) for all x₂, x₃
X₁ = (X₁, X₂ = x₂, X₄ = x₄) for all x₂, x₄
X₁ = (X₁, X₃ = x₃, X₄ = x₄) for all x₃, x₄
:
https://math.stackexchange.com/questions/3269920/ https://math.stackexchange.com/questions/1920473/
independent and identically distributed process
原本定義是 i.i.d. = identically distributed & mutually independent 但是有一個定理是 i.i.d. => stationary 因此實際上是 i.i.d. = stationary & mutually independent
https://www.statlect.com/glossary/stationary-sequence
moment / coherence
definition: moment: statistics of one random variable coherence: statistics of more random varibles property: stationary <=> constant moment & constant coherence mutually independent => zero coherence (e.g. uncorrelated) property: indentically distributed => constant moment stationary <=> constant moment & constant coherence pairwise independent => zero first-order coherence mutually independent => zero higher-order coherence i.i.d. => constant moment & zero coherence
nth moment
E[Xⁿ]
nth central moment
E[(X-X̄)ⁿ] where X̄ = E[X] is mean
first-order coherence
E[XY]
first-order standardized coherence
E[(X-X̄)(Y-Ȳ)]
———————————————————————
(E[(X-X̄)²]E[(Y-Ȳ)²])¹⸍²
higher-order coherence
E[(XY,YZ,XZ)] C(3,2)
E[(AB,AC,AD,BC,BD,CD)] C(4,2)
moment: mean central moment: zero, variance, skewness coherence: correlation standardized coherence: correlation coefficient
Cauchy distribution: moment and coherence is undefined 處處連續處處不可微分，處處平滑處處不可解析，凡事總有例外。
ergodic process
sample / ensemble
數字number：一個數字。 隨機變數random variable：一個浮動數字，擁有浮動範圍。既非隨機、亦非變數。 觀測值observation：一個數字，迎合給定的隨機變數。 取樣sampling：生成一個／多個觀測值，迎合給定的隨機變數。 樣本sample：承上，那些觀測值。形成索引集。
數列sequence：一串數字。 隨機過程stochastic process：一串浮動數字，各自擁有浮動範圍。 樣本函數sample function：一串數字，迎合給定的隨機過程。 集成ensembling：生成一個／多個樣本函數，迎合給定的隨機過程。 集體ensemble：承上，那些樣本函數。形成索引集。
stochastic process: {x0, x1, x2, ..., x7}
x0 = U{0,1}
x2 = U{0,1}
x1 = U{0,1}
: :
x7 = U{0,1}
s0 = { 1 3 2 1 5 4 2 4 } │ sample
s1 = { 0 1 1 0 1 1 0 1 } │ functions
s2 = { 1 0 1 1 1 0 0 1 } ↓
──────────────→
time
ergodic process
ergodic process:
ensemble average = time average
s0 = { 1 0 0 1 0 1 0 0 } │ ensemble
s1 = { 0 1 1 0 1 1 0 1 } │ average
s2 = { 1 0 1 1 1 0 0 1 } ↓
──────────────→
time average
(1) ensemble average
calculate expected value E[] with multiple sample functions
(2) time average
calculate expected value E[] with only one sample function
多次實驗的統計值，改成單次實驗的統計值，節省計算時間。
stationary => ergodic
stochastic process calculus
stability
definition: 1. stable: difference of two sequences is bounded 2. convergent: sequence approaches constant (steady state) 3. stationary: sequence is constant theorem: convergent => stationary at steady states (talking about a trivial thing like a theorem)
for LTI system, theorem: 1. stable: sequence is bounded 2. stable <=> Re[pᵢ] < 0 <=> convergent 3. convergent: sequence approaches zero (steady state) 大家習慣省略主角convergent。喪心病狂。 1. stable => stationary at steady states 2. stable <=> Re[pᵢ] < 0 (people pretend not to see 'convergent') (textbooks replace all 'convergent' with 'stable')
weakly stationary process
專著《Stochastic Systems for Engineers: Modelling, Filtering and Control》。
weakly stationary process (wide-sense stationary process)
教科書習慣採用weakly stationary達成time-invariant autocorrelation。 但是(1)是多餘的，成為歷史共業。
identically distributed process = time-invariant distribution stationary process = time-invariant joint distribution
stationary process: (1) time-invariant moment (mean/variance/skewness/...) (2) time-invariant coherence (autocorrelation/...) stationary process example: 1. i.i.d. process 2. steady states
weakly stationary process: (1) time-invariant 1st moment (mean) (2) time-invariant 1st coherence (autocorrelation) 性質更弱。失去了高階動差和高階相干。 property: (3) time-invariant 2nd central moment (variance) (1) & (2) => (3)。證明省略。也有人將(3)直接納入定義。 weakly stationary process example: 1. white noise 2. random phase cosine wave
古人曾經區分wide-sense和weakly，給予不同定義。 但是這種定義沒有充分理解stationary，也似乎沒有實際用途。 wide-sense stationary process: (1) time-invariant 1st moment (mean) (2) time-invariant 1st coherence (autocorrelation) weakly stationary process: (1) time-invariant 1st moment (mean) (2) time-invariant 2nd moment (variance) property: strictly => wide-sense => weakly
correlation
專著《Modeling, Identification and Simulation of Dynamical Systems》。
correlation
兩道訊號，取兩個時間點相乘。 autocorrelation function: φxx[n₁,n₂] = x[n₁] x[n₂] cross-correlation function: φxy[n₁,n₂] = x[n₁] y[n₂] 推廣為複數，後者取共軛複數。 φxy[n₁,n₂] = x[n₁] y[n₂] 推廣為MIMO，改成點積。 φxy[n₁,n₂] = x[n₁] ∙ y[n₂] 推廣為隨機系統，取期望值。 φxy[n₁,n₂] = E[x[n₁] ∙ y[n₂]]
time-invariant correlation
簡化成位移量k = n₂ - n₁。位置n = n₁可以是任意數值。 φxx[k] = E[x[n+k] x[n]] φxy[k] = E[x[n+k] y[n]] 減法風格 φxx[k] = E[x[n] x[n-k]] φxy[k] = E[x[n] y[n-k]] 減法風格要特別小心加總運算／期望值運算的起點索引值。 如果起點索引值是從零開始，那麼就會遭遇負索引值。 我個人不推薦減法風格。 (personally, I do not recommend this.) (negative index appears when summate from index 0)
property: (1) φxy ≠ φyx cross-correlation is not commute (2) φxy = -⃡φyx useless since negative input is not defined property for LTI system: (3) φxx ∗ f = φyx where f is LTI system (4) -⃡f ∗ φxx ∗ f = φyy where f is LTI system
ergodic process:
多次實驗的平均數，改成滑動視窗的平均數，節省計算時間。
sliding window
n₁ n₂
|--> |-->
experiment 1: ~~~~~~~~~~~~~~~ |
experiment 2: ~~~~~~~~~~~~~~~ | ensemble average
: V
-------------->
time average
Rxx[k] = 1/N sum { x[n+k] x[n] }
n=0⋯N-1
Rxy[k] = 1/N sum { x[n+k] y[n] }
n=0⋯N-1
雜訊通常是非時變相關。 輸入訊號、輸出訊號通常不是非時變相關。 常見的非時變相關訊號，例如常數函數、餘弦函數。出現於穩態。 系統識別當中， 時域最小平方估計當中， 自相關函數／互相關函數不是非時變相關！ 輸入訊號可以是任意訊號！ 大家習慣想成那是另外一套定義，只不過數學式子恰好一模一樣。 那套定義主要用途是簡化數學式子、簡化描述方式。
correlation spectrum
autocorrelation spectrum Φxx[ω] = sum { φxx[k] exp(-𝑖ωk) }
cross-correlation spectrum Φxy[ω] = sum { φxy[k] exp(-𝑖ωk) }
k=-∞⋯+∞
正式名稱如下。名稱太長了，很少人使用。 autocorrelation spectral density function cross-correlation spectral density function 統計學與訊號學當中， spectral density function是指定義域是頻域的函數（統計而得）。 大家習慣簡稱spectral density，省略後綴function。 其中spectral是指定義域是頻域。 其中density function是指機率密度函數（積分等於一、已經正規化）。 但是大家不一定有做正規化。 spectrum也是定義域是頻域的函數。 因此大家常常將spectral density改稱為spectrum。 畢竟大家不一定有做正規化。無傷大雅。 甚至有人省略主角correlation。喪心病狂。縮短後名稱如下： auto spectral density / cross-spectral density autospectrum / cross-spectrum
cross-correlation spectrum 是指dtft(xcorr(x,y)) spectral cross-correlation 是指xcorr(dtft(x),dtft(y)) bilinear的情況下，兩者應該相等吧？【尚待確認】 dtft(xcorr(x,y)) = xcorr(dtft(x),dtft(y))
power spectrum
energy spectrum = squared amplitude spectrum power spectrum = squared amplitude spectrum / N energy／power差別在於沒做正規化／有做正規化。
Wiener–Khinchin theorem: autocorrelation spectrum = power spectrum periodogram: 使用autocorrelation spectrum，作為power spectrum的估計值。 time-invariant autocorrelation: 此定理的前提是兩個時間點t1 t2可以簡化成一個時間差。 完全沒有必要使用weakly stationary process。 但是教科書喜歡使用weakly stationary process。成為歷史共業。 integrability and boundness: 此定理的前提是傅立葉轉換必須存在。 傅立葉轉換是積分變換，必須滿足可積性。 可積性細分許多條件，其中包含受限性。 受限性：輸出受限（不會是正負無限大）。 積分變換的情況下：輸入受限，而且在無限遠處趨近零。輸出亦然。 即是autocorrelation approaches zero at infinite lag。 autocorrelation spectrum亦然。 stable system: 穩定的LTI FIR system的impulse response會變成零。 穩定的LTI IIR system的impulse response會趨近零。 進而滿足autocorrelation approaches zero at infinite lag。
範例
ARX(1) model:
e
╷
+↓
u ───┬──────┴──────┬─────→ y
+↑ ┌───────┐
│ │ delay │
│ ┌────┐ └───────┘
└──│ ×a │←────┘
└────┘
y[k] = a y[k-1] + u[k] + e[k]
impulse response (with zero-mean white noise):
┌──────────┐ │ e ~ WN(0,σ²)
│ 1 │ ↓
u ───→│ ———————— │─────→ y
│ 1 - az⁻¹ │
└──────────┘
1
y[k] = ———————— u[k]
1 - az⁻¹
y[k] = a y[k-1] + u[k]
autocorrelation (with zero-mean white noise):
Ryy[t] = σ² pow(a,|t|) (0 < a < 1)
σ² (1 - a²) σ² (1 - a²)
Φyy[ω] = ———————————————— = ——————————————————
|1 - a exp(-𝑖ω)|² 1 + a² - 2a cos(ω)
(Φyy is white iff a = 0)
Ryy[t] Φyy[ω]
| | __
1 | /\ | / \
| / \ | _- -_
| ____-- --_____ | __-- --___
|--------|----------> t |--------|----------> ω
0 0
(the peak is slightly higher than 1)
example:
respiration (low frequency) (slow)
ECG (high frequency) (fast)
Rxx[t]
| __
1 | __--/\--__
| __-- / \ --__slow
| _____-- --_____fast
0 |---------|----------> t
0
延伸閱讀
stochastic process
https://tcs.nju.edu.cn/wiki/index.php?title=概率论与数理统计_(Spring_2025)
martingale
一個隨機過程，最新的K個隨機變數，期望值是定值。
https://en.wikipedia.org/wiki/Doob_martingale https://en.wikipedia.org/wiki/Azuma's_inequality When analyzing sums, random walks, or other additive functions of independent random variables, one can often apply the central limit theorem, law of large numbers, Chernoff's inequality, Chebyshev's inequality or similar tools. When analyzing similar objects where the differences are not independent, the main tools are martingales and Azuma's inequality. Azuma's inequality applied to the Doob martingale gives the method of bounded differences (MOBD) which is common in the analysis of randomized algorithms.
concentration
一個隨機變數，實際取樣集中於平均數附近。
https://en.wikipedia.org/wiki/Concentration_inequality https://en.wikipedia.org/wiki/Concentration_of_measure
generated by ChatGPT concentration inequality Markov's inequality: Gives an upper bound on the probability that a non-negative random variable exceeds a certain value. Chebyshev's inequality: Provides a bound on how much a random variable deviates from its mean, in terms of its variance. Chernoff bounds: Provide exponentially decreasing bounds on the tail probabilities of sums of independent random variables. Hoeffding's inequality: Gives a concentration bound for the sum of bounded independent random variables.
concentration of measure Chernoff Bound: Gives exponential decay bounds for the tail probabilities of sums of independent random variables, often used in concentration of measure results. McDiarmid's Inequality: Provides concentration bounds for functions of independent random variables, stating that a function of independent variables is unlikely to deviate too far from its expected value if each variable has only a small effect on the function. Gaussian concentration: For Gaussian distributions, the concentration of measure shows that the probability that a Gaussian random variable deviates by more than a fixed amount from its mean is exponentially small.
correlation / causation
conditional probability 只關注某一塊子集合 代入身分建立視點
Bayes's theorem 切換視點
independence 即便到了子集合裡面比例也一樣，換句話說，在宇集合內很均勻
p_xy(x,y) = p_x|y(x,y) p_y(y) = p_x(x) p_y(y)
p_x|y(x,y) = p_x(x)
correlation 正比反比關係
causation 因果關係
association 上述所有東西的泛稱
deterministic system / random system
deterministic system = evolution rule is fixed random system = evolution rule is random 函數／系統，分成兩種。 公式是固定的、公式不是固定的。
deterministic stochastic process random stochastic process 數字推廣成分布。 浮動數列也是分成兩種。 然而大家目前尚未認知到這件事。 浮動數列的數字的數學公式。 確定的：數字有明確數學公式。甚至使用了前後數字。 隨機的：數字亂七八糟。甚至看不出是否使用了前後數字。
noise🚧
noise
「雜訊」或「噪訊」。雜訊沒有明確定義。目前大家認為雜訊包含幾種意義：一、不被需要的訊號、冗餘的訊號。二、沒有規律的訊號、不可預測的訊號。三、無法擬合數學模型的冗餘訊號。
各種真實現象擁有各種雜訊，例如聲音、語音、電流、電波。大家觀察各種真實現象，命名為各種雜訊。大家也利用隨機過程建立數學模型，創造出各種雜訊。維基百科整理了一份列表：
noise的頻譜
大家仿照光譜由紅到紫的特性，嘗試分類雜訊。
white: 振幅為常數 grey: 振幅符合人類聽覺曲線。（不那麼白） red: 振幅正比於頻率倒數平方。 以頻率對數為座標軸，漸減6dB。 pink: 振幅正比於頻率倒數。 以頻率對數為座標軸，漸減3dB。（不那麼紅） violet: 振幅負正比於頻率倒數平方。 以頻率對數為座標軸，漸增6dB。 blue: 振幅負正比於頻率倒數。 以頻率對數為座標軸，漸增3dB。（不那麼紫）
noise
noise
大家習慣假設： 一、雜訊不隨時間而變。 二、雜訊的平均數是零、變異數是常數。
Gaussian noise N(μ,σ²) uniform noise U(a,b)
noise: 1. stationary process 2. uncorrelated process Gaussian noise: 1. stationary process 2. mutually independent process => uncorrelated white noise: 1. stationary process 2. uncorrelated process 3. white (power spectrum is uniform distribution)
Gaussian noise / white noise
Gaussian noise: for any time interval, the joint distribution is always Gaussian distribution (time domain) white noise: for any time interval, the power spectrum is always uniform distribution (frequency domain)
高斯雜訊：每個訊號都是常態分布。 白雜訊：強度頻譜平方（能量頻譜）是常數函數（均勻分布）。
Gaussian noise
Gaussian random variable
高斯隨機變數：一個浮動數字，浮動範圍是高斯分布（常態分布）。 簡單起見，大家不稱作高斯隨機變數，而是稱作高斯分布（常態分布）。
常態分布擁有許多強悍的數學性質： 1. reproductive property / closure property 兩個常態分布，相加是常態分布。 多個常態分布，線性組合是常態分布。 2. Lévy–Cramér theorem 相加是常態分布，且兩個分布獨立，則是兩個常態分布。 線性組合是常態分布，且這些分布互相獨立，則是個個常態分布。 3. Kac–Bernstein theorem / Darmois–Skitovitch theorem 相加、相減，恰好獨立，則是兩個常態分布。 兩種線性組合，恰好獨立，則是個個常態分布。 4. central limit theorem 多個相同分布，而且互相獨立，相加得到一個分布。 其平均數也是一個分布，而且趨近常態分布。
Gaussian process
高斯過程：一串浮動數字，任意子集合的聯合分布皆是高斯分布。 高斯過程：一串浮動數字，聯合分布是聯合高斯分布。
高斯隨機變數、高斯過程擁有許多強悍的數學性質： 1. 高斯分布，independent <=> uncorrected。變成雙向。 2. 高斯過程，stationary <=> weakly stationary。變成雙向。 3. 高斯過程，可以做微積分，稱作Itô calculus。 數字推廣成分布，一切事物維持運作，那就只能是高斯分布。
Gaussian noise
高斯誤差：高斯過程。
white noise
white noise
white noise in frequency domain:
power spectrum is uniform <=> no 1st coherence
(pairwise independent => no 1st coherence)
white noise in time domain:
never discovered
a sufficient but not necessary condition:
weakly stationary process with Wiener–Khinchin theorem
white noise in theory:
(1) time-invariant 1st moment (mean)
(2) time-invariant 1st coherence (autocorrelation)
(3) autocorrelation approaches zero at infinite lag
其中(2)和(3)足以實施逆向傅立葉轉換。
white noise in practice:
(1) constant mean
E[x[n]] = μ for all n
(1&2) constant variance
Var[x[n]] = σ² for all n
(2&3) impulse autocorrelation (uncorrelated)
Rx[t] = σ² δ[t]
實務上無視高階動差，只處理mean和variance。
實務上無視高階相干，只處理autocorrelation，並且視作不相關。
white noise的頻譜
隨機過程，實施「傅立葉轉換」，從時域變頻域。
普通的隨機過程：無解析解，頻譜混亂。
weakly stationary process：有解析解。兩兩的共相關數，可求得頻譜。具備傅立葉轉換的相關數學特性，諸如線性、卷積乘法對偶、能量守恆。
然而，weakly stationary process，就是每個數字幾乎一樣的數列。缺乏討論意義，也無法解決現實問題。數學家目前僅發現weakly stationary process，尚未發現更具討論意義的隨機過程。
noise reduction（noise cancellation）（denoising）
「雜訊降低」或「降噪」或「去雜訊」。分離並消除雜訊。
各種真實現象擁有各種雜訊，例如聲音、語音、電流、電波。各種雜訊各有專門的雜訊降低方法、各有專門的書籍。由於這不是我的專長，就不多提了。
disturbance rejection
「干擾拒絕」。分離並消除系統雜訊。
系統雜訊稱作干擾。輸入訊號或輸出訊號或同時擁有雜訊。干擾通常是高斯白雜訊、其平均數為零、其變異數是常數。由於這不是我的專長，就不多提了。