Incremental learning is a machine learning approach where a model learns from new data while retaining previous knowledge. It allows the model to adapt and improve its performance over time without retraining from scratch. It is useful for handling streaming data, dynamic environments, and lifelong learning scenarios. Incremental learning involves updating the model incrementally as new data becomes available, enabling the accumulation of knowledge while adapting to changing conditions.
Import packages
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
import numpy as npGenerate dummy data
This is the random generated data we will use for the incremental learning demontration
X = np.random.rand(100, 1)
y = 2 * X.squeeze() + np.random.randn(100)
print("X:", X)
print("y:", y)Output: X: [[0.76176646] [0.69613846] [0.9877934 ] [0.4808156 ] [0.02509009] [0.40655198] .......... [0.63026455] [0.87792955] [0.0773053 ] [0.3727879 ] [0.87910821] [0.72120154] [0.11854516] [0.43418629]] y: [ 1.26389604 0.04846767 2.51986081 0.54058779 0.25143496 -0.49453571 0.79937518 0.17758522 1.26013815 -0.38571676 0.9934525 0.60253407 1.46893879 0.8380549 2.96598813 1.07231094 -0.14663261 0.04888111 3.43871512 2.27246165 1.92397542 1.60956402 0.61986814 -2.7380499 -1.44152951 0.25838784 -0.17440325 -1.26192062 1.77675232 1.80807172 1.65887447 0.8093043 -2.04504172 1.47508602 1.32883494 -0.42027407 0.49399519 0.58326674 2.79088189 -0.13388995 2.57109613 0.88019958 0.44556527 -1.06384853 0.27849504 1.34932125 0.78117022 2.46683493 0.69511834 -0.31952751 1.46631497 0.97256006 -0.13129591 1.06934562 2.23650218 0.73386191 0.19257869 0.70229706 1.14358587 -0.82606993 1.2464345 0.72815756 0.87469108 -0.79811047 1.37172085 0.592716 1.11722417 1.44352933 2.0594432 -0.00481194 0.81536972 0.09204644 0.7246622 2.36635355 1.08829129 2.53989606 1.21484815 0.33005071 0.49195937 1.44010814 2.71279204 -0.90303375 1.02039572 1.76583169 2.15780203 0.41748203 2.64011397 1.02914845 0.64881031 -0.38511983 2.35708319 1.95689564 1.72807686 0.03624875 0.25913553 1.36682549 2.13488535 1.40792715 -3.13275323 -0.1823195 ]
Split the data
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print("X_train size:", X_train.size)
print("X_test size:", X_test.size)
print("y_train size:", y_train.size)
print("y_test size:", y_test.size)Output: X_train size: 80 X_test size: 20 y_train size: 80 y_test size: 20
Model
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html
# Create an instance of SGDRegressor
model = SGDRegressor(learning_rate='constant', eta0=0.01, random_state=42)
print("model:",model)
model: SGDRegressor(learning_rate='constant', random_state=42)Incremental learning
# Perform incremental learning
for i in range(X_train.shape[0]):
print("X:",X_train[i])
x = X_train[i].reshape(1, -1)
print("Reshaped X:",x)
y_true = y_train[i]
print("y_true:",y_true)
# Update the model with a single sample
model.partial_fit(x, [y_true])
# Make predictions on the test set after each iteration
y_pred = model.predict(X_test)
# Calculate the mean squared error
mse = mean_squared_error(y_test, y_pred)
print(f"Iteration {i+1} - MSE: {mse:.4f}")
print("===============================================")
Here we are doing Icremental Learning. We are picking the single sample and training our model with Partial_fit().
model.partial_fit(x, [y_true])
It allows model to preserve previous knowledge and also increase its knowledge with help of new sample data.
we are incrementaly training our model 100 times in small 100 step in this for loop.
After every learing step we are predicting the outcome and checking the MSE.
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
In below output we can clearly observe. after every next batch of incremental learning model performance is improving and MSE is getting reduced gradualy.
Output:
X: [0.74166183] Reshaped X: [[0.74166183]] y_true: 0.733861909381499 Iteration 1 - MSE: 2.4492 =============================================== X: [0.51577494] Reshaped X: [[0.51577494]] y_true: 0.6488103110814958 Iteration 2 - MSE: 2.4274 =============================================== X: [0.09255618] Reshaped X: [[0.09255618]] y_true: -0.17440324565158588 Iteration 3 - MSE: 2.4326 =============================================== X: [0.8434641] Reshaped X: [[0.8434641]] y_true: 0.44556526846404165 Iteration 4 - MSE: 2.4159 =============================================== X: [0.10448319] Reshaped X: [[0.10448319]] y_true: -0.004811943427009857 Iteration 5 - MSE: 2.4165 =============================================== X: [0.74472979] Reshaped X: [[0.74472979]] y_true: 1.0723109388750867 Iteration 6 - MSE: 2.3775 =============================================== X: [0.86978645] Reshaped X: [[0.86978645]] y_true: 2.5710961264972836 Iteration 7 - MSE: 2.2803 =============================================== X: [0.87910821] Reshaped X: [[0.87910821]] y_true: 2.134885345519915 Iteration 8 - MSE: 2.2033 =============================================== X: [0.31375557] Reshaped X: [[0.31375557]] y_true: -0.385716758955279 Iteration 9 - MSE: 2.2171 =============================================== X: [0.59754964] Reshaped X: [[0.59754964]] y_true: 0.7246621994777648 Iteration 10 - MSE: 2.1965 =============================================== X: [0.40529778] Reshaped X: [[0.40529778]] y_true: 0.6025340713538603 Iteration 11 - MSE: 2.1814 =============================================== X: [0.94950155] Reshaped X: [[0.94950155]] y_true: 2.4668349314235565 Iteration 12 - MSE: 2.0944 =============================================== X: [0.0088946] Reshaped X: [[0.0088946]] y_true: 0.41748202886282304 Iteration 13 - MSE: 2.0871 =============================================== X: [0.29650848] Reshaped X: [[0.29650848]] y_true: 1.7767523200116087 Iteration 14 - MSE: 2.0424 =============================================== X: [0.87792955] Reshaped X: [[0.87792955]] y_true: 0.03624875390569815 Iteration 15 - MSE: 2.0482 =============================================== X: [0.40655198] Reshaped X: [[0.40655198]] y_true: -0.4945357120624121 Iteration 16 - MSE: 2.0668 =============================================== X: [0.39166631] Reshaped X: [[0.39166631]] y_true: 1.1172241702009347 Iteration 17 - MSE: 2.0393 =============================================== X: [0.53903265] Reshaped X: [[0.53903265]] y_true: 0.5927160015243202 Iteration 18 - MSE: 2.0266 =============================================== X: [0.3825592] Reshaped X: [[0.3825592]] y_true: -0.42027407181389487 Iteration 19 - MSE: 2.0429 =============================================== X: [0.31002787] Reshaped X: [[0.31002787]] y_true: -0.14663260866680905 Iteration 20 - MSE: 2.0510 =============================================== X: [0.43904951] Reshaped X: [[0.43904951]] y_true: -0.3195275115455667 Iteration 21 - MSE: 2.0649 =============================================== X: [0.40397714] Reshaped X: [[0.40397714]] y_true: 1.328834935570797 Iteration 22 - MSE: 2.0312 =============================================== X: [0.4627018] Reshaped X: [[0.4627018]] y_true: 0.17758521530201554 Iteration 23 - MSE: 2.0310 =============================================== X: [0.3727879] Reshaped X: [[0.3727879]] y_true: 1.3668254926599128 Iteration 24 - MSE: 1.9975 =============================================== X: [0.85752356] Reshaped X: [[0.85752356]] y_true: -1.2619206150144138 Iteration 25 - MSE: 2.0489 =============================================== X: [0.84998013] Reshaped X: [[0.84998013]] y_true: 2.2724616478598407 Iteration 26 - MSE: 1.9774 =============================================== X: [0.22719348] Reshaped X: [[0.22719348]] y_true: -0.9030337463845763 Iteration 27 - MSE: 2.0046 =============================================== X: [0.10197954] Reshaped X: [[0.10197954]] y_true: 0.2583878351367551 Iteration 28 - MSE: 2.0019 =============================================== X: [0.86745882] Reshaped X: [[0.86745882]] y_true: 0.8746910845299726 Iteration 29 - MSE: 1.9794 =============================================== X: [0.15406104] Reshaped X: [[0.15406104]] y_true: 0.8380548951963447 Iteration 30 - MSE: 1.9629 =============================================== X: [0.26792812] Reshaped X: [[0.26792812]] y_true: -1.4415295117002749 Iteration 31 - MSE: 2.0048 =============================================== X: [0.4808156] Reshaped X: [[0.4808156]] y_true: 0.5405877895954401 Iteration 32 - MSE: 1.9944 =============================================== X: [0.05175575] Reshaped X: [[0.05175575]] y_true: 0.048881106302082915 Iteration 33 - MSE: 1.9965 =============================================== X: [0.93462433] Reshaped X: [[0.93462433]] y_true: 2.7908818869113183 Iteration 34 - MSE: 1.9073 =============================================== X: [0.49211954] Reshaped X: [[0.49211954]] y_true: 1.2601381456950356 Iteration 35 - MSE: 1.8782 =============================================== X: [0.19573672] Reshaped X: [[0.19573672]] y_true: 0.491959367637739 Iteration 36 - MSE: 1.8711 =============================================== X: [0.3370274] Reshaped X: [[0.3370274]] y_true: 0.7993751808362578 Iteration 37 - MSE: 1.8562 =============================================== X: [0.82386826] Reshaped X: [[0.82386826]] y_true: 1.3717208500273042 Iteration 38 - MSE: 1.8217 =============================================== X: [0.06004781] Reshaped X: [[0.06004781]] y_true: 0.49399519462585534 Iteration 39 - MSE: 1.8154 =============================================== X: [0.55984396] Reshaped X: [[0.55984396]] y_true: -0.38511983003878814 Iteration 40 - MSE: 1.8339 =============================================== X: [0.7773255] Reshaped X: [[0.7773255]] y_true: 0.19257868784382492 Iteration 41 - MSE: 1.8371 =============================================== X: [0.43418629] Reshaped X: [[0.43418629]] y_true: -0.18231949667311143 Iteration 42 - MSE: 1.8486 =============================================== X: [0.27278893] Reshaped X: [[0.27278893]] y_true: 2.2365021804317244 Iteration 43 - MSE: 1.7993 =============================================== X: [0.53938194] Reshaped X: [[0.53938194]] y_true: -1.063848527810815 Iteration 44 - MSE: 1.8367 =============================================== X: [0.43963111] Reshaped X: [[0.43963111]] y_true: 1.4663149700081548 Iteration 45 - MSE: 1.8043 =============================================== X: [0.29195797] Reshaped X: [[0.29195797]] y_true: 1.4435293288199667 Iteration 46 - MSE: 1.7751 =============================================== X: [0.59650604] Reshaped X: [[0.59650604]] y_true: 0.7811702222451266 Iteration 47 - MSE: 1.7616 =============================================== X: [0.95481754] Reshaped X: [[0.95481754]] y_true: 2.05944319764618 Iteration 48 - MSE: 1.7070 =============================================== X: [0.32440081] Reshaped X: [[0.32440081]] y_true: 0.7281575597689751 Iteration 49 - MSE: 1.6965 =============================================== X: [0.72120154] Reshaped X: [[0.72120154]] y_true: 1.4079271525048964 Iteration 50 - MSE: 1.6667 =============================================== X: [0.33512596] Reshaped X: [[0.33512596]] y_true: 1.4401081390325203 Iteration 51 - MSE: 1.6403 =============================================== X: [0.05292784] Reshaped X: [[0.05292784]] y_true: 0.8801995816185462 Iteration 52 - MSE: 1.6283 =============================================== X: [0.02064604] Reshaped X: [[0.02064604]] y_true: 1.143585866743488 Iteration 53 - MSE: 1.6116 =============================================== X: [0.71715126] Reshaped X: [[0.71715126]] y_true: 0.6951183434733902 Iteration 54 - MSE: 1.6035 =============================================== X: [0.11854516] Reshaped X: [[0.11854516]] y_true: -3.132753229924153 Iteration 55 - MSE: 1.6739 =============================================== X: [0.42098196] Reshaped X: [[0.42098196]] y_true: 0.7022970573129523 Iteration 56 - MSE: 1.6644 =============================================== X: [0.90768464] Reshaped X: [[0.90768464]] y_true: 2.5398960558733794 Iteration 57 - MSE: 1.6005 =============================================== X: [0.09622874] Reshaped X: [[0.09622874]] y_true: -2.045041716798856 Iteration 58 - MSE: 1.6474 =============================================== X: [0.0773053] Reshaped X: [[0.0773053]] y_true: 0.2591355258161889 Iteration 59 - MSE: 1.6474 =============================================== X: [0.08479052] Reshaped X: [[0.08479052]] y_true: -0.8260699314640254 Iteration 60 - MSE: 1.6695 =============================================== X: [0.45177031] Reshaped X: [[0.45177031]] y_true: -0.7981104741916505 Iteration 61 - MSE: 1.6976 =============================================== X: [0.78347805] Reshaped X: [[0.78347805]] y_true: 2.157802028459123 Iteration 62 - MSE: 1.6457 =============================================== X: [0.3838709] Reshaped X: [[0.3838709]] y_true: 0.5832667433639784 Iteration 63 - MSE: 1.6394 =============================================== X: [0.84600224] Reshaped X: [[0.84600224]] y_true: 1.8080717154168062 Iteration 64 - MSE: 1.5988 =============================================== X: [0.69613846] Reshaped X: [[0.69613846]] y_true: 0.048467671125916434 Iteration 65 - MSE: 1.6086 =============================================== X: [0.30175467] Reshaped X: [[0.30175467]] y_true: -0.13129590612794273 Iteration 66 - MSE: 1.6187 =============================================== X: [0.1243288] Reshaped X: [[0.1243288]] y_true: 1.6095640215806917 Iteration 67 - MSE: 1.5916 =============================================== X: [0.9877934] Reshaped X: [[0.9877934]] y_true: 2.5198608149791 Iteration 68 - MSE: 1.5305 =============================================== X: [0.14291422] Reshaped X: [[0.14291422]] y_true: -2.7380499020549727 Iteration 69 - MSE: 1.5916 =============================================== X: [0.61994482] Reshaped X: [[0.61994482]] y_true: 1.0291484502619161 Iteration 70 - MSE: 1.5752 =============================================== X: [0.43265339] Reshaped X: [[0.43265339]] y_true: 1.9568956376061468 Iteration 71 - MSE: 1.5379 =============================================== X: [0.4744713] Reshaped X: [[0.4744713]] y_true: 1.088291285252918 Iteration 72 - MSE: 1.5214 =============================================== X: [0.81181296] Reshaped X: [[0.81181296]] y_true: 2.6401139720693187 Iteration 73 - MSE: 1.4635 =============================================== X: [0.99292492] Reshaped X: [[0.99292492]] y_true: 1.0203957162744621 Iteration 74 - MSE: 1.4506 =============================================== X: [0.75990505] Reshaped X: [[0.75990505]] y_true: 1.9239754233941646 Iteration 75 - MSE: 1.4148 =============================================== X: [0.67762428] Reshaped X: [[0.67762428]] y_true: 1.2464344952023867 Iteration 76 - MSE: 1.3971 =============================================== X: [0.56436766] Reshaped X: [[0.56436766]] y_true: 0.09204644153717556 Iteration 77 - MSE: 1.4061 =============================================== X: [0.56338703] Reshaped X: [[0.56338703]] y_true: 2.9659881327062365 Iteration 78 - MSE: 1.3506 =============================================== X: [0.63026455] Reshaped X: [[0.63026455]] y_true: 1.7280768635481303 Iteration 79 - MSE: 1.3239 =============================================== X: [0.42590363] Reshaped X: [[0.42590363]] y_true: 0.9725600601015791 Iteration 80 - MSE: 1.3143 ===============================================
Evaluation
Lets check how well our incremental learning model works. We are using Mean Squared Error[MSE] as metrics to evaluate the model.
# Final evaluation on the test set
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print(f"Final MSE: {mse:.4f}")Final MSE: 1.3143
Watch full Video tutorial
