📐 PCA: Dimensionality Reduction

Mengompresi banyak fitur menjadi sedikit — tanpa membuang informasi penting.

🎬 Animasi: PCA: Sumbu Variansi Terbesar

Sumbu PC1 berputar mencari arah variansi maksimum, lalu data diproyeksikan ke sana → dimensi turun tanpa kehilangan banyak informasi.

📌 1. Konsep: Arah Variansi Maksimal

Data sering "menyebar" di arah tertentu (korelasi antar fitur). PCA menemukan sumbu baru di mana data paling menyebar — lalu memproyeksikan data ke sumbu-sumbu itu.

Data 2D yang memanjang diagonal:

        •
      • •          ← data menyebar kuat di arah diagonal
    • •
  • •
  •

PCA:
• PC1 = arah variansi TERBESAR  (diagonal di atas)
• PC2 = arah variansi terbesar berikutnya (tegak lurus PC1)

Proyeksi ke PC1 saja (1 dimensi) → kehilangan informasi MINIMAL.

Algoritma:
1. Center data (mean = 0)
2. Hitung matriks kovarians  C = (1/(n-1)) · ZᵀZ
3. Cari eigenvector C dengan eigenvalue terbesar → PC1
   (di sini: power iteration + deflation)
4. Ulangi untuk PC2, PC3, ...

"Variance explained" = eigenvalue / total eigenvalue
→ berapa % informasi yang tersimpan di tiap komponen.

📌 2. Contoh: PCA pada Data Berkorelasi

print("=== PCA: DATA 2D BERKORELASI ===\n")
random_seed(42)
n = 100
base = [randn() for _ in range(n)]
# fitur 1 dan 2 berkorelasi kuat (memanjang diagonal)
X = [[2.0 * b + 0.3 * randn(), 1.0 * b + 0.3 * randn()] for b in base]

print("Sebelum PCA: data memanjang diagonal")
plot_points(X, None, title="Data asli (2D)")

def pca(X, n_components=2):
    n = len(X); d = len(X[0])
    mu = [mean([row[j] for row in X]) for j in range(d)]
    Z = [[row[j] - mu[j] for j in range(d)] for row in X]
    C = [[sum(Z[i][a] * Z[i][b] for i in range(n)) / (n - 1)
          for b in range(d)] for a in range(d)]
    comps, vars_ = [], []
    M = [row[:] for row in C]
    for _ in range(n_components):
        v = [rand() for _ in range(d)]
        s = math.sqrt(sum(x * x for x in v))
        v = [x / s for x in v]
        for _ in range(300):
            Mv = [sum(M[a][b] * v[b] for b in range(d)) for a in range(d)]
            s = math.sqrt(sum(x * x for x in Mv))
            if s < 1e-12:
                break
            v = [x / s for x in Mv]
        Mv = [sum(M[a][b] * v[b] for b in range(d)) for a in range(d)]
        eig = dot(v, Mv)
        comps.append(v); vars_.append(eig)
        for a in range(d):
            for b in range(d):
                M[a][b] -= eig * v[a] * v[b]
    return Z, comps, vars_

Z, comps, vars_ = pca(X, 2)
total = sum(vars_)
print("\nKomponen utama:")
for i, (v, var) in enumerate(zip(comps, vars_)):
    print("  PC%d: arah (%+.2f, %+.2f)  variansi = %.1f%%"
          % (i + 1, v[0], v[1], var / total * 100))

# Proyeksi ke 1D (PC1 saja)
proj1 = [dot(z, comps[0]) for z in Z]
print("\nProyeksi 1D (PC1): range [%.2f .. %.2f], std=%.2f"
      % (min(proj1), max(proj1), std(proj1)))
print("→ 1 angka ini menyimpan %.1f%% variansi data." % (vars_[0] / total * 100))

proj2 = [[dot(z, comps[0]), dot(z, comps[1])] for z in Z]
plot_points(proj2, None, title="Setelah PCA: PC1 mendatar, PC2 menegak")

Output:

💡 Perhatikan: setelah PCA, PC1 dan PC2 tidak berkorelasi (kovarians antar komponen = 0). Fitur baru saling independen — ini bagus untuk algoritma yang mengasumsikan independensi.

📌 3. Contoh: Kompresi 3D → 2D

Kasus nyata: data 100 fitur → diproyeksikan ke 2-3 dimensi untuk visualisasi, atau ke 10 dimensi untuk mempercepat training.

print("=== PCA: 3D → 2D ===\n")
random_seed(42)
n = 100
# Data 3D: 2 dimensi informatif + 1 dimensi "hampir noise"
X3 = [[randn(), randn(), 0.05 * randn()] for _ in range(n)]

var_orig = sum(variance([row[j] for row in X3]) for j in range(3))
print("Variansi total data 3D: %.3f" % var_orig)
for j in range(3):
    print("  fitur %d: variansi = %.3f" % (j, variance([row[j] for row in X3])))

def pca(X, n_components=2):
    n = len(X); d = len(X[0])
    mu = [mean([row[j] for row in X]) for j in range(d)]
    Z = [[row[j] - mu[j] for j in range(d)] for row in X]
    C = [[sum(Z[i][a] * Z[i][b] for i in range(n)) / (n - 1)
          for b in range(d)] for a in range(d)]
    comps, vars_ = [], []
    M = [row[:] for row in C]
    for _ in range(n_components):
        v = [rand() for _ in range(d)]
        s = math.sqrt(sum(x * x for x in v))
        v = [x / s for x in v]
        for _ in range(300):
            Mv = [sum(M[a][b] * v[b] for b in range(d)) for a in range(d)]
            s = math.sqrt(sum(x * x for x in Mv))
            if s < 1e-12:
                break
            v = [x / s for x in Mv]
        Mv = [sum(M[a][b] * v[b] for b in range(d)) for a in range(d)]
        eig = dot(v, Mv)
        comps.append(v); vars_.append(eig)
        for a in range(d):
            for b in range(d):
                M[a][b] -= eig * v[a] * v[b]
    return Z, comps, vars_

Z, comps, vars_ = pca(X3, 2)
total = var_orig
print("\nVariansi yang dijelaskan:")
for i, var in enumerate(vars_):
    print("  PC%d: %.1f%%" % (i + 1, var / total * 100))

proj = [[dot(z, comps[0]), dot(z, comps[1])] for z in Z]
var_proj = sum(variance([p[j] for p in proj]) for j in range(2))
print("\nVariansi setelah proyeksi 2D: %.3f (%.1f%% dari total)"
      % (var_proj, var_proj / total * 100))
print("→ Fitur ke-3 (noise) dibuang hampir tanpa rugi.")

plot_points(proj, None, title="Proyeksi 2D (fitur noise dibuang)")

Output:

⚠️ PCA adalah unsupervised: ia tidak peduli label. Arah variansi terbesar belum tentu arah yang paling memisahkan kelas. Untuk itu ada supervised alternative: LDA (Linear Discriminant Analysis). Dan seperti semua metode berbasis jarak — scaling dulu!

🧠 Latihan

Ubah noise fitur ke-3 dari 0.05 menjadi 1.0. Berapa variansi yang dijelaskan PC1 sekarang?
Hitung korelasi antar koordinat proyeksi (PC1 vs PC2) di contoh 2. Harusnya ≈ 0 — buktikan.
Buat data 4D, proyeksikan ke 2D. Berapa % variansi yang tersimpan?
Jelaskan kenapa PCA bisa dipakai untuk denoising: proyeksi ke k komponen lalu "rekonstruksi" kembali.