By the end of this, you will have a solid grasp on this algorithm — helpful for both interview preparation and day-to-day work as a DS.

Quick Pause: If you’re new here Subscribe — my goal is to make Data Science easy for you. 👇🏻

Subscribe now

What is a Decision Tree?

The algorithm uses a tree-like structure for decisions to either predict the target value (regression) or predict the target class (classification).

Before diving into how decision trees work, let us become familiar with the basic structure and terminologies of a decision tree:

• Root Node: The topmost node representing all data points.

• Splitting: It refers to dividing a node into two or more sub-nodes.

• Decision Node: Nodes further split into sub-nodes; a split node.

• Leaf / Terminal Node: Nodes that do not split; final results.

• Branch / Sub-Tree: Subsection of the entire tree.

• Parent and Child Node: Parent node divides into sub-nodes; children are the sub-nodes.

• Pruning: Removing sub-nodes of a decision node is called pruning. Pruning is often done in decision trees to prevent overfitting.

What Is a Decision Tree Regressor?

A Decision Tree Regressor is used when the target is continuous — such as predicting a house price or stock value.

It learns the input-output relationship by breaking the dataset into smaller segments.
At each node, the algorithm chooses a feature and a split point that minimizes the prediction error.

How It Works

• The algorithm observes feature values.

• It builds a tree that predicts continuous outputs.

• At each split, it tries to reduce the mean squared error (MSE) (or other evaluation metrics.

• The process continues until certain stopping criteria are met.

How is Decision Tree Classifier Different?

Classification trees are used to predict categorical data (yes, no), while regression trees are used to predict numerical data, such as the price of a stock.

Classification and regression trees are powerful tools for analyzing data.

Training Decision Trees

A decision tree in general parlance represents a hierarchical series of binary decisions. Including,

• Letting the algorithm find the best split at each level.

• Building branches until either:

  • The tree reaches a maximum depth, or

  • The data in a node is pure enough, or

  • A minimum number of samples is reached.

Rather than setting rules manually, the algorithm figures out the optimal conditions for splits on its own.

Do not forget to explore the full implementation, including code and data: Github Repository 👈🏻

Alright, that’s a wrap! If you’ve made it this far — thank you! Stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

But real-world datasets, we often have:

• Too many features.

• Correlated predictors.

• Noise, Outliers and much more.

That is why we need regularization.

Let’s break down what it is, different types of regularization techniques, and how to choose between them like a data scientist who knows what they're doing.

And hey — if you’re new here Subscribe, as my goal is to simplify Data Science for you. 👇🏻

Subscribe now

Let’s dive in!

Regularization in Machine Learning

It is a statistical method to reduce errors caused by overfitting on training data, by adding a penalty term to the cost function in the model.

This discourages complex models with high coefficients, promoting simpler and more generalizable solutions.

Regularization helps improve a model's performance on unseen data and enhances its overall robustness.

Techniques of Regularization

There are mainly two types of regularization techniques, which are given below:

1. Ridge Regression (L2 Regularization)

2. Lasso Regression (L1 Regularization)

each influencing the model's behavior in different ways.

1. Ridge Regression (or L2 Regularization)

Ridge Regression adds a squared penalty to the coefficients:

• RSS is the residual sum of squares.

• λ (lambda) controls the strength of the penalty.

What It Does:

• Keeps all features in the model, but shrinks coefficients toward zero.

• Especially helpful when predictors are correlated.

• Doesn’t eliminate features—just balances them.

Use Ridge When:

• You want to keep all variables, but reduce model complexity.

• Multicollinearity is an issue.

• Interpretability isn’t your top priority.

2. Lasso Regression (or L1 Regularization)

Lasso uses the absolute value of coefficients:

What It Does:

• Shrinks some coefficients exactly to zero.

• Performs feature selection automatically.

• Helps build sparse models—great for high-dimensional datasets.

Use Lasso When:

• You want to select important features and ignore the rest.

• Your dataset has many variables, but not all of them matter.

• You care about simpler, interpretable models.

Bonus: ElasticNet

Why choose between Ridge and Lasso when you can have both?

ElasticNet combines both L1 and L2 penalties:

Best for:

• High-dimensional data.

• When predictors are correlated.

Note: Elastic Net seems to perform empirically better which combines both the above methods.

Here’s a visual comparison between Ridge Regression (L2), Lasso Regression (L1), and Elastic Net (L1 + L2), showing how each method constrains the coefficient estimates during regularization, as discussed above.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

Stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

1. Feature Extraction Component (Turning Text into Numbers):

   1. This part takes a piece of text and turns it into a set of features or characteristics (basically, numbers).

   2. These features help the system understand what's important in the text.

2. Classifier/ Regressor Component (Making a Decision):

   1. Once the features are generated, this part of the system uses them to decide what category label or fine-grained scores the text belongs to.

   2. It matches the features with a list of known categories/ scores and assigns the most appropriate one to the text.

By the end of this read, you will have a solid understanding of:

• What Text Vectorization is?

• Why it is essential while working with Text data? and,

• What popular techniques you can use?

This is helpful for day-to-day work as a Data Scientist.

And hey — if you’re new here Subscribe, as my goal is to simplify Data Science for you. 👇🏻

Subscribe now

Let’s dive in!

What is Feature Extraction from Text or Text Vectorization?

Machine learning algorithms require numerical input.

Hence, we required to use Text Vectorization to converts textual data into numerical format, enabling algorithms to process and analyze it effectively.

Why do we need Feature Extraction from Text?

Text vectorization captures important information from sequential text data.

It includes word frequency, relationships between words, semantic and syntactic meaning.

Simply stating,

• Which words are common?

• Which words are rare?

• How words are connected?

• What the sentence might mean?

All of this helps the machine learn and make better predictions.

Text Feature Extraction Techniques

Here's a comparison table of common Text Feature Extraction Techniques used in Natural Language Processing (NLP).

Tips for Use:

Want to try these techniques? Here’s how to choose:

• Choose Word2Vec / FastText / GloVe for capturing semantic relationships.

• Use BERT or Transformer-based models when context is critical (e.g., QA, NER).

👇🏻 Additionally you can also checkout:

Stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

Today, let’s understand the core differences in how these algorithms work.

And how they relate to two powerful ML algorithms: Random Forest (bagging) and XGBoost (boosting).

Make sure to hit that Subscribe button, so you don’t miss when I put out any new posts!

Subscribe now

In machine learning, ensemble methods combine multiple models to improve performance.

Two of the most popular ensemble techniques are bagging and boosting.

Understanding their differences is crucial to choosing the right model for your problem.

Let's explore!

🌲 What is Bagging?

Bagging stands for Bootstrap Aggregating.

The idea is simple:

• It builds multiple models (usually decision trees) on different random subsets of the data.

• These subsets are created through sampling with replacement (i.e. bootstrap sampling).

• Predictions are averaged (for regression model) or voted (for classification model).

Random Forest is the most used bagging algorithm. It builds many decision trees and merges them to get a more accurate and stable prediction.

🚀 What is Boosting?

Boosting is a sequential technique:

• It builds models one after another, where each new model focuses on the mistakes of the previous one.

• Instead of sampling, it adjusts the weights of the training data points.

• The final prediction is a weighted sum of all models.

XGBoost (also called, Extreme Gradient Boosting) is a highly efficient and scalable boosting algorithm known for its accuracy and speed.

Here’s a side-by-side comparison:

Both Random Forest and XGBoost are powerful algorithms:

• Use Random Forest if you want a fast, robust, low bias-low variance model that’s easy to tune and less prone to overfitting.

• Use XGBoost if you need maximum predictive power and are ready to spend time tuning the parameters.

In practice, it’s common to try both and compare performance using cross-validation.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

Until next time, happy learning!

— Nikita Prasad

That’s how you build models that are smart, simple, and ready for real-world problems.

Let’s find out how can you stop a Gradient Boosting Model from overfitting.

Make sure to hit that Subscribe button, so you don’t miss when I put out any new posts!

Subscribe now

What is Gradient Boosting Algorithm?

It is an ensemble technique that combines the predictions of multiple weak learners, typically decision trees, sequentially to create a single and more accurate strong learner.

It can be used for classification and regression tasks. 👈

Advantages of Gradient Boosting Model:

• Robustness to missing values and outliers

• Effective handling of high cardinality categorical features

How to prevent Overfitting in GBM?

Let’s discuss three important hyper-parameters provided by Sklearn to stop a Gradient Boosting model from overfitting.

1. Using Fewer Trees

In Gradient Boosting, you build models called trees one after another.

• If you build too many trees, the model can become too complex.

• Building fewer trees can help keep the model simple and general.

You can control the number of trees by setting a number using n_estimators parameter.

from sklearn.ensemble import GradientBoostingClassifier

gbm = GradientBoostingClassifier(n_estimators=20)

Here, you tell the model to build only 20 trees.

2. Make Trees Shorter

Shorter trees mean the model learns simple patterns, not complicated noise.

You can control how tall or deep the trees are by setting max_depth.

gbm = GradientBoostingClassifier(max_depth=3)

A depth of 3 means the tree is not allowed to ask too many questions before making a decision.

3. Slow Down the Learning

Another way to prevent overfitting is to make the model learn slowly.

You can do this by setting a smaller learning_rate.

gbm = GradientBoostingClassifier(learning_rate=0.1)

If the learning rate is too high, the model can jump too quickly to conclusions.

But small learning rate makes the model take small careful steps.

Tip: If you lower the learning rate, you might need more trees to reach good performance.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

Until next time, happy learning!

— Nikita Prasad

It’s simple, intuitive and easy to compute.

But is it enough?

In many cases — especially when dealing with imbalanced data — accuracy alone can be misleading.

Let’s go through other reliable evaluation metrics in this read, in a simple and clear way.

If you’re new here Subscribe as here, I want to make Data Science easy for you. 👇🏻

Subscribe now

What is Accuracy?

This metric that tells you how often the classifier model is correct.

You can calculate it as:

Accuracy = (Number of Correct Predictions) ÷ (Total number of Predictions

In Python, using accuracy_score, you can predict the accuracy of your model.

print(accuracy_score(test_target,y_pred)) # Output : 0.853

This means the model was right 85.3% of the time.

But, When Accuracy Can Be Misleading?

Accuracy fails when classes are imbalanced.

In cases where one class dominates the dataset, a classifier might achieve high accuracy by simply predicting the dominant class for all instances.

I’ve already covered a high-level overview of How to Handle Imbalanced Datasets. Missed it? (really?) Go check it out. 👇🏻

This is exactly why data scientists must know other evaluation metrics.

What is Confusion Matrix?

Simply stating, it's a table, that describes the performance of a classification model.

It presents a breakdown of the correct and incorrect predictions by each class.

Why Confusion Matrix Should Be Used?

As it provides more insights than accuracy alone.

It allows you to see where the model is making errors, such as confusing one class with another.

It’s also plays a vital role in designing effective A/B tests and accurately interpret the results.

In case you missed the Most Important Interview Question! Find it here: 👇🏻

So… Can You Calculate Accuracy from the Confusion Matrix?

Yes, you can.

You just have to sum up the correct predictions (True Positives and True Negatives) and divide by the total number of predictions.

where:

• True Positives (TP): It is the case where you predicted Yes and the real output was also Yes.

• True Negatives (TN): It is the case where you predicted No and the real output was also No.

• False Positives (FP): It is the case where you predicted Yes but it was actually No.

• False Negatives (FN): It is the case where you predicted No but it was actually Yes.

Note: But, the opposite is not possible. You cannot create a full Confusion Matrix just by knowing the Accuracy, because accuracy is just a number.

What is Precision?

It measures the accuracy of the positive predictions made by the classifier.

Here’s it’s formula:

Say, if the classifier predicts Yes 10 times, and only 7 were actually correct, then the precision is 7 out of 10, or 0.7.

What is Recall or Sensitivity?

Recall tells you how many actual positive cases your model was able to identify.

Here’s it’s formula:

Basically, it answers the question: “Out of all the real Yes cases, how many did the model find?”

Trade-Off between Precision & Recall

Often there’s a trade-off between precision and recall.

Meaning increasing one can decrease the other. Which one to prioritize depends on the application.

Many real-world applications requires a balance—this is where the F1 Scores comes in.

What is F1 Score?

F1 score is the harmonic mean of precision and recall.

It's useful when you want to find a balance between precision and recall, especially when classes are imbalanced.

Why Harmonic Mean?

Because it punishes extreme values more.

And, if either precision or recall is very low, the F1 Score will be low, helpful with imbalance classes.

What is Specificity?

While recall tell you how good the model is at identifying positives, specificity tells us how good it is at identifying negatives.

Useful in domains where detecting negatives correctly is just as important as positives.

Classification Report

Scikit-learn provides a classification report that summarizes all key metrics for each class, using the following code.

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))

This typically includes:

• Precision,

• Recall,

• F1 score,

• Support (the number of actual samples for each class in the specified dataset).

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

Understanding these metrics helps data scientists make informed decisions about model performance, fairness, and suitability for real-world use.

Thanks for reading!

Until next time, happy learning!

— Nikita Prasad

I would highly recommend you to give it a read, if you haven’t till now 👇🏻:

And while you’re at it, subscribe to me so you’ll never miss any more of these reads.

Subscribe now

What is Causal Self-Attention Mechanism?

Causal or Masked self-attention ensures that the model's prediction for a certain position in a sequence is only dependent on the known previous outputs, and not on future tokens.

How to Hide Future Tokens with Causal Attention?

In causal attention, the attention weights above the diagonal are masked.

This is to ensure that for any given input, the LLM is unable to utilize future tokens while calculating the context vectors with the attention weight.

To achieve this, for each given token, we mask out the future tokens (the ones that come after the current token in the input text).

The simplest way is to mask out the unnormalized attention scores above the diagonal with negative infinity before they enter the softmax function.

mask = torch.triu(torch.ones(context_length, context_length), diagonal=1)
masked = attn_scores.masked_fill(mask.bool(), -torch.inf)
print(masked)

"""
Output: 
tensor([[0.0988,   -inf,   -inf,   -inf,   -inf],
        [0.1345, 0.1951,   -inf,   -inf,   -inf],
        [0.1330, 0.1957, 0.2100,   -inf,   -inf],
        [0.1985, 0.2716, 0.2985, 0.1392,   -inf],
        [0.0688, 0.1003, 0.1079, 0.0486, 0.1117]],
       grad_fn=<MaskedFillBackward0>)
"""

Note: The attention weights in each row correctly sum to 1.

Masking Additional Attention Weights With Dropout

In addition, we also apply dropout to reduce overfitting during training of the LLM.

Dropout is a DL technique where randomly selected hidden layer units are ignored during the training of the Neural Network for preventing overfitting and improving generalization.

In GPT Models, dropout can be applied in several places, such as:

• after computing the attention weights

• or, after multiplying the attention weights with the value vectors

Though, it is recommended to apply the dropout mask after computing the attention weights.

Furthermore, in this specific example, we use a dropout rate of 50%, which means randomly masking out half of the attention weights.

Later while training the GPT model, we will use a lower dropout rate, such as 0.1 or 0.2.

If we apply a dropout rate of 0.5 (50%), the non-dropped values will be scaled accordingly by a factor of:

1 / (1 - dropout_rate) = 1/0.5 = 2

This scaling is crucial to maintain the overall balance of the attention weights, ensuring that the average influence of the attention mechanism remains consistent during both the training and inference phases.

Implementing CausalAttention Class

Now, we are ready to implement a self-attention class, including the causal and dropout masks.

Step 1: Compared to the previous SelfAttention class, we added a dropout layer.

Step 2: The register_buffer call is also a new addition.

Step 3: Transpose dimensions 1 and 2, keeping the batch dimension at the first position (i.e., 0).

class CausalAttention(nn.Module):

    def __init__(self, d_in, d_out, context_length, dropout, qkv_bias=False):
        super().__init__()
        self.d_out = d_out
        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_key   = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.dropout = nn.Dropout(dropout)
        self.register_buffer('mask', torch.triu(torch.ones(context_length, context_length), diagonal=1))

    def forward(self, x):
        b, num_tokens, d_in = x.shape # New batch dimension b
        keys = self.W_key(x)
        queries = self.W_query(x)
        values = self.W_value(x)

        attn_scores = queries @ keys.transpose(1, 2) # Changed transpose
        attn_scores.masked_fill_(  # Note `_` operations are in-place in PyTorch
            self.mask.bool()[:num_tokens, :num_tokens], -torch.inf)  # `:num_tokens` to account for cases where the number of tokens in the batch is smaller than the supported context_size
        attn_weights = torch.softmax(
            attn_scores / keys.shape[-1]**0.5, dim=-1
        )
        attn_weights = self.dropout(attn_weights) 

        context_vec = attn_weights @ values
        return context_vec

Note: In PyTorch, operations with a trailing underscore (_) are performed in-place, avoiding unnecessary memory copies.

Instantiating CausalAttention Class:

print(d_in)
print(d_out)

torch.manual_seed(123)
# Instantiating CausalAttention Class
context_length = batch.shape[1]
ca = CausalAttention(d_in, d_out, context_length, 0.0)

context_vecs = ca(batch)

print(context_vecs)
print("context_vecs.shape:", context_vecs.shape)

"""
Output: 
3
2
tensor([[[-0.3585,  0.0415],
         [-0.4451,  0.0252],
         [-0.4656,  0.0393],
         [-0.5461, -0.0500],
         [-0.5129, -0.0367]],

        [[-0.3585,  0.0415],
         [-0.4451,  0.0252],
         [-0.4656,  0.0393],
         [-0.5461, -0.0500],
         [-0.5129, -0.0367]]], grad_fn=<UnsafeViewBackward0>)
context_vecs.shape: torch.Size([2, 5, 2])
"""

As we can see, the resulting context vector is a 3D tensor where each token is now represented by a 2D embedding.

Note: Dropout is only applied during training, not during inference.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

And that’s a wrap!

Next, we will expand on this concept and implement a multi-head attention module, that implements several of such causal attention mechanisms in parallel.

If you’ve made it this far — thank you so much, stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

Keeping up with best coding practices using PyTorch can give you a significant edge in this competitive market.

In this read, I’ve discussed 5 practical tips to improve your PyTorch codes!

Want to learn “WHY” and “HOW” of ML, Gen-AI, Data Science & Analytics and much more? It’s all here. 👇🏻

Subscribe now

But the first thing to note is that,

Why PyTorch?

Internet is full of debate over PyTorch vs. Tensorflow. While both have their respective pros and cons.

PyTorch stands out for its flexiblity, and widespread adoption in both academia and industry.

With its Dynamic Computation Graph and powerful GPU acceleration, PyTorch makes deep learning development seamless and efficient.

That being said, let’s jump into the core of this article—

How to write better PyTorch code?

Here are 5 practical tips that can surely elevate your coding practices:

1. Utilizing DataLoaders for Streamlining Data Processing

While working with large datasets, manually loading data using loops can slow down the training process.

Instead, you can use PyTorch’s DataLoader, which efficiently handles batching, shuffling and parallel loading for you while utilizing multiprocessing for speed.

import torch
from torch.utils.data import DataLoader, TensorDataset

# Creating a sample dataset
data = torch.randn(1000, 3)  # 1000 samples, 3 features
labels = torch.randint(0, 2, (1000,))  # Binary labels

# Converting dataset to PyTorch dataset (TensorDataset)
dataset = TensorDataset(data, labels)

# Use DataLoader for batching and shuffling the data
data_loader = DataLoader(dataset, batch_size=32, shuffle=True, num_workers=4)

for batch in data_loader:
    input_batch, target_batch = batch
    print(input_batch.shape, target_batch.shape)  # Outout: (32,3) (32,)

2. Using GPU Acceleration Efficiently

One of the biggest advantages of PyTorch is its seamless support for GPUs.

However, many beginners forget to transfer their models and data to the GPU, which can result in slow CPU-based computation.

So, here’s the proper way to utilize your GPU:

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model.to(device)  # Move model to GPU
data, labels = data.to(device), labels.to(device)  # Move data to GPU

Additionally, you can also check for GPU’s memory usage:

print(torch.cuda.memory_summary())

This helps you avoid memory overflow issues when working with large models.

3. Using Automatic Mixed Precision (AMP) for Faster Training

PyTorch provides native support for Mixed Precision Training, allowing you to speed up model training while reducing memory usage.

from torch.cuda.amp import GradScaler, autocast

scaler = GradScaler()

for data, labels in data_loader:
    optimizer.zero_grad()
    with autocast():  # Enables mixed precision
        outputs = model(data)
        loss = loss_fn(outputs, labels)
    
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

Note: GradScaler or Gradient scaling helps prevent gradients with small magnitudes from flushing to zero (“underflowing”) when training with mixed precision.

Mixed Precision Training can lead to a 2-3x increase in speed while maintaining the same level of accuracy.

4. Implementing Checkpoints for Long Training Sessions

You probably know that DL models can take hours (or even days) to train.

So, to prevent losing progress due to unexpected shutdowns, it is recommended to always save model checkpoints at regular intervals.

# Saving Model
torch.save(model.state_dict(), "model_checkpoint.pth")

# Loading Model
model.load_state_dict(torch.load("model_checkpoint.pth"))

Additionally, you can also consider saving the optimizer state for resuming training seamlessly, using below code snippet:

# Saving Optimizer State
torch.save({
    "epoch": epoch,
    "model_state_dict": model.state_dict(),
    "optimizer_state_dict": optimizer.state_dict(),
    "loss": loss,
}, "checkpoint.pth")

5. Profile Your Code for Performance Optimization

PyTorch also provides you with a built-in profiler that helps detect bottlenecks in your code, such as slow data loading, inefficient computations, or memory leaks.

You can use this code:

from torch.profiler import profile, record_function, ProfilerActivity

# Profiling the Code
with profile(activities=[ProfilerActivity.CPU], ProfilerActivity.CUDA,                record_shapes=True) as prof:
    with record_function("model_inference"):
        model(data)

print(prof.key_averages().table(sort_by = "cpu_time_total", row_limit=10))

This can help you optimize performance by identifying the most time-consuming parts of your code.

As an apiring data scientist using these tips will not only help you streamline your deep learning workflows but also enhance your productivity.

Bonus: Read about how to know the best value of your hyperparmeters to utilize Full Potential of your Computationally EXPENSIVE Neural Networks! 👇🏻

Got more cool PyTorch tips? Comment Down 👇

Until next time, happy learning!

— Nikita Prasad

YES, it degrades over time!

Machine learning models are trained with historical data.

But when deployed in the real-world, they encounter live data that constantly evolves.

As the environment changes, models can become outdated and lose accuracy over time—a phenomenon called model drift.

So, let’s dive deeper into why this happens and how you can prevent model decay.

⏸️ Quick Pause: If you’re new here, I’d highly appreciate if you subscribe to receive bi-weekly data tips and insights — directly into your inbox. 👇🏻

Subscribe now

How Your Predictions Can FAIL?

Imagine you’re a data scientist working on cab fare prediction model.

Your goal is to ensure that predicted fares closely match real-world pricing.

Your model likely relies on input features such as:

• Distance of the ride,

• Time of day (peak vs. non-peak),

• Traffic conditions.

• Weather conditions, etc.

Everything works fine—until one day, your product manager rushes in with a drastic issue: fares are way off!

Despite careful training and validation, the model’s accuracy has plummeted.

What’s Going Wrong?

After ruling out any data quality issues, two usual suspects emerge: data drift and concept drift—two critical challenges in production ML system.

So, it’s important to understand the difference between them because they require different approaches to detect and fix.

What is Data Drift?

Data drift happens when the statistical distribution of input data changes.

And this shift leads to unreliable predictions because the model encounters unfamiliar patterns.

Example:

Your cab fare model was trained on normal traffic patterns. But, new highways open, drastically changing travel times.

Now, the model’s predictions are unreliable.

How to Detect and Fix Data Drift?

Detection Methods:

1. Feature Distribution Monitoring: Compare real-time input distributions with historical training data using statistical tests like Kolmogorov-Smirnov (KS) or Population Stability Index (PSI).

2. Drift Metrics: Track changes in mean, variance, and percentiles of key input features over-time.

3. Out-of-Distribution (OOD) Detection: Use anomaly detection models to flag inputs that differ significantly from training data.

Mitigation Strategies:

• Frequent Model Retraining: Periodically update the model using the latest data.

• Adaptive Models: Implement online learning techniques that allow models to adapt dynamically to new data distributions.

What is Concept Drift?

Unlike data drift, concept drift occurs when the relationship between features and the target changes.

Practically stating, what you’re trying to predict has changed.

Example

Say your cab fare model was trained on a per-mile pricing system. But stakeholders decided to switch to a flat-rate fare structure.

The input features remain the same, but their relationship to fare price has changed—making the model’s predictions inaccurate.

How to Detect and Fix Concept Drift

Detection Methods:

1. Feature-Target Relationship Analysis: Continuously track correlations between input variables (e.g., distance, time) and the target variable (fare price).

2. Error Distribution Monitoring: If prediction errors systematically increase over time, it may indicate concept drift.

3. Model Comparison: Maintain a simple baseline model; if its performance surpasses the production model, concept drift may be at play.

Mitigation Strategies:

• Business Rule Awareness: Work closely with product teams to stay updated on evolving pricing policies.

• Hybrid Approaches: Combine rule-based adjustments with ML models to accommodate dynamic pricing structures.

REMEMBER: To prevent silent model degradation, always question

• has the data has changed?

• has the business logic has shifted?

• is the model is still aligned with reality?

By actively monitoring for data and concept drift, retraining models and collaborating with domain experts, you can stay ahead of model decay before it impacts business decisions.

Comment Down 👇: Have you ever faced model degradation? How did you handle it?

Until next time, happy learning!

— Nikita Prasad

And while you’re at it, Subscribe me so as you’ll not miss any more of these contents.

Subscribe now

In this read, let’s gear up to improve the model by adding some trainable weights, just like those used in the original transformer architecture, the GPT models, and most other popular LLMs.

Note: This Self-Attention Mechanism is also called "Scaled Dot-Product Attention".

Here’s the overall idea (similar to before):

• Computing context vectors as weighted sums over the input vectors, specific to a certain input element.

• For this, you need attention weights (normalized attention scores that sum up to 1, using the softmax function).

Here’s the modified architecture: 👇🏻

You can realise, there’re only slight differences compared to the basic attention mechanism, introduced earlier:

• The most notable difference is the introduction of weight matrices (W_q, W_k, and W_v) that are updated during model training.

  • These trainable weight matrices are crucial, so that the model (specifically, the attention module inside the model) can learn to produce reliable context vectors.

• Also, now you have to scale the attention scores by dividing them by the square root of the embedding dimension, √d_k (i.e., d_k**0.5):

So, that being said let’s get into the coding, and explore more!

Implementing Self-Attention with Trainable Weights

Let’s me start by introducing the three training weight matrices: W_q, W_k, and W_v.

These three matrices are used to project the embedded input tokens, x_i, into query, key, and value vectors via. matrix multiplication:

• Query vector: q_i =W_qx_i

• Key vector: k_i =W_kx_i

• Value vector: v_i = W_vx_i

The embedding dimensions of the input x and the query vector q can be the same or different, depending on the model's design and specific implementation.

import torch

input_emb = torch.tensor([
    [0.12, 0.45, 0.67],  # "Attention"
    [0.34, 0.56, 0.78],  # "Mechanism"
    [0.23, 0.57, 0.91],  # "drives"
    [0.76, 0.88, 0.45],  # "contextual"
    [0.54, 0.12, 0.34]   # "embedding"
], dtype=torch.float32)

Note: In GPT models, the input and output dimensions are usually the same.

But for illustration purposes, we’re using choosing different input and output dimensions to better follow the computation:

d_in = input_emb.shape[1] # the input embedding size, d=3
d_out = 2                 # the output embedding size, d=2

Following steps are implemented in the below code snippet:

Step 1: Convert Input to Query, Key, and Value Vectors: Multiply input embedding by weight matrices (Wq, Wk, Wv) to get queries (Q), keys (K), and values (V).

Step 2: Compute Attention Scores: Take the dot product of Queries and Keys to measure how much each word should focus on others.

Step 3: Scale the Scores: Divide by square root of embedding size (√dₖ) to keep values stable.

Step 4: Apply Softmax: Convert scores into probabilities so they sum to 1 (higher score = more focus).

Step 5: Compute Context Vectors: Multiply attention weights with Value vectors (V) to get the final context representation.

Step 6: Return Context Vectors: These final context vectors are returned and will be used in further layers of the GPT model.

import torch.nn as nn

class SelfAttention(nn.Module):

    def __init__(self, d_in, d_out, qkv_bias=False):
        super().__init__()
        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_key   = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)

    def forward(self, x):
        keys = self.W_key(x)
        queries = self.W_query(x)
        values = self.W_value(x)
        
        attn_scores = queries @ keys.T
        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)

        context_vec = attn_weights @ values
        return context_vec

Note: Instead of manually defining trainable weight matrices, using PyTorch's Linear Layers (nn.Linear) has a preferred weight initialization scheme, which leads to more stable model training.

Let's instantiate a tokenizer object of SelfAttention class and tokenize our sampled input text:

torch.manual_seed(123)
sa = SelfAttention(d_in, d_out)
print(sa(input_emb))

"""
Output: 
tensor([[-0.5128, -0.0366],
        [-0.5141, -0.0376],
        [-0.5143, -0.0377],
        [-0.5143, -0.0377],
        [-0.5129, -0.0367]], grad_fn=<MmBackward0>)
"""

Bravo! You have successfully implemented a self-attention mechanism.

However, we’ve just scraped the tip of the iceberg.

Want to go deeper? Subscribe for upcoming deep dives into GPT Architectures!

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

And that’s a wrap! If you’ve made it this far — thank you so much, stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

And as a data scientist, your job is to extract meaningful insights from data.

But, biases can silently creep into the models, leading to inaccurate predictions and poor decision-making.

So, let’s explore 6 most critical biases you must watch out for and how to handle them.

⏸️ Quick Pause: If you’re new here, I’d highly appreciate if you subscribe to recieve bi-weekly data tips and insights — directly into your inbox. 👇🏻

Subscribe now

1. Correlation vs. Causation Fallacy – Not Everything That Moves Together Is Connected

A company notices that employees who wear headphones are more productive.

But does wearing headphones cause productivity, or are highly focused employees more likely to wear headphones?

This is the classic scenario of confusing correlation (two things happening together) with causation (one thing directly causing the other).

How to avoid it?

• Always try to answer, Could there be third factor influencing both variables?

• Use controlled experiments (e.g., A/B testing) to establish causation.

BONUS READ: 👇🏻

2. Simpson’s Paradox – When Trends Trick You

Imagine you’re analyzing hospital performance and find that Hospital A has a higher survival rate than Hospital B.

Seems simple and clear, right?

But when you break it down by patient severity, Hospital B actually has better survival rates for both mild and critical cases.

What happened?

This is called Simpson’s Paradox.

When data is grouped incorrectly, your trend might be positive at the global level, but when you break it down, the trends may reverse.

The problem arises because different sub-groups may behave differently.

How to avoid it?

• Always segment your data to check if trends hold across different groups.

• Look at confounding variables—hidden factors that could explain the results.

3. Survivorship Bias – The Data You Don’t See Matters

During World War II, engineers studied bullet holes on returning planes to decide where to add armor.

They initially thought to reinforce the areas with the most bullet holes.

But they were missing a key insight—planes hit in other areas never returned.

This is survivorship bias—focusing only on successful cases while ignoring failures.

How to avoid it?

• Consider thinking out of the box like what’s missing from the dataset, before drawing conclusions.

• Look at failure cases as well, not just success stories.

4. Selection Bias – The Wrong Sample, the Wrong Conclusions

Imagine you analyze customer feedback and find that 95% of users love your app.

You conclude that your product is performing exceptionally well.

However, your data only includes responses from active users—those who already enjoy the app.

The problem?

You're missing feedback from users who churned or never engaged in the first place.

If you only analyze feedback from happy users, you might overestimate customer satisfaction and miss critical pain points that drive users away.

This is selection bias—when your sample doesn’t represent the entire population.

How to avoid it?

• Use random sampling to ensure diverse representation.

• Be mindful of who is excluded from the dataset.

5. Confirmation Bias – Seeing What You Want to See

People who believe in a conspiracy theory tend to mostly read sources that support it.

That’s confirmation bias—favoring information that confirms your existing beliefs while ignoring contradictory evidence.

In data science, this happens when you test models but only report metrics that support your hypothesis.

How to avoid it?

• Challenge your assumptions—ask, What would prove me wrong?

• Perform A/B testing and validate results using multiple methods.

6. Omitted Variable Bias – The Missing Piece of the Puzzle

A study finds that students who drink coffee score higher on exams.

But it ignores the fact that students who drink coffee might also study more.

Omitted variable bias happens when you ignore an important factor that affects both the cause and effect.

How to avoid it?

• Identify all possible influences before drawing conclusions.

• Use domain expertise and statistical tests to check for missing variables.

I hope this guide helped you gain a deeper understanding of biases and their impact on building robust models also produce accurate, fair insights.

REMEMBER:

• to always question your data,

• validate your assumptions,

• look for bigger picture rather than what’s easy to measure, and

• be wary of drawing conclusions from observational data alone.

Comment Down 👇: Have you ever made a decision based on data that later turned out to be misleading?

Until next time, happy learning!

— Nikita Prasad

However, a model isn’t truly reliable unless it meets key statistical assumptions.

Linear regression relies on several assumptions to ensure the validity and reliability of the model's results.

In this read, let’s explore the 5 Main Assumptions of Linear Regression, in detail.

Before that, if you’re new here Subscribe, as my goal is to simplify Data Science for you. 👇🏻

Subscribe now

Now, let’s get started!

Assumptions of Linear Regression:

These assumptions are often asked in Data Science interviews, so understanding them is crucial!

1. Linearity Among Dependent and Independent Variables

For linear regression to work, the relationship between the independent variables (predictors) and the dependent variable (response) should be approximately linear.

This means that the change in the response variable should be proportional to changes in the predictor variables.

How to check?

You can verify this assumption by examining:

• Scatter plots: Check if the data points form a straight-line pattern.

  

• Residual plots: Ensure there is no curved pattern.

2. No Multicollinearity in the Data

Multicollinearity occurs when one predictor variable is highly correlated with one or more other predictor variables.

This makes it difficult for the model to determine which variable is actually influencing the dependent variable.

How to check?

• Variance Inflation Factor (VIF): Measures how much a predictor’s variance is increased by multicollinearity. If VIF > 5 or 10, it indicates high multicollinearity.

  

• Correlation Matrix: Identifies highly correlated predictor variables using the .corr method of a Pandas series.

Important Note: Just because two features are correlated doesn’t mean one causes the other. This is known as Correlation vs Causation Fallacy.

While this may seem obvious, computers can't really differentiate between correlation and causation, why is why human insight is required.

3. Normality of Residuals (or Errors)

The residual (errors) should be normally distributed.

Note: While this is not required for estimating regression coefficiets, it is crucial for hypothesis testing and constructing confidence intervals.

How to check?

Normality of errors can be checked using techniques such as:

• Kernel density estimation (KDE): A smoothed representation of the residual distribution.

  

• Q-Q Plot: If residuals align with the straight reference line, they are normally distributed.

  

• Shapiro-Wilk Test: A statistical test for normality.

4. Homoscedasticity Among the Data

Homoscedasticity means that the variance of the errors should remains the same for all values of the predictors (independent variables).

If the variance changes (gets bigger or smaller), the assumption is violated, and this is called heteroscedasticity (not preferable).

How to check?

• Residual plots: Residuals should be randomly scattered. If there is a funnel shape, it indicates heteroscedasticity.

Since, in above plot the spread of the residuals doesn't have any significant increases or decreases with the test predictor, it suggests homoscedasticity in data.

5. No Auto-correlation of Residuals

Auto-correlation happens when the reiduals are related to each other over time.

This usually occurs in time-series data where past values influence future values.

Why is this bad?

• If residuals are correlated, the model underestimates the standard error, making predictions unreliable.

How to check?

• Durbin-Watson test: A statistical test for detecting autocorrelation.

• Residual plot over time: Look for patterns instead of randomness.

  

Clearly, above plot shows is no pattern or correlation present in the residual points.

If there is no pattern, autocorrelation is not present.

And that’s a wrap, whether you’re a beginner diving into machine learning or an expert, understanding these assumptions will help you build better models.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository. 👈🏻

Also stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

Knowing how to deal with it not only helps you tackle interview questions but also enables you to build better predictive models.

Quick Pause: If you’re new here Subscribe — my goal is to make Data Science easy for you. 👇🏻

Subscribe now

What Is Imbalanced Data?

Imbalanced data refers to a dataset within which one or some of the classes (or labels) make up the majority of the data set, leaving far fewer examples of others.

This issue applies to both classification and regression tasks.

• In classification, it might happen in binary classification, multi-class classification and multi-label classification.

  • For instance, a dataset where 95% of the data belongs to class ”blue” and the rest 5% belongs to the other class “orange” (see the plot below).

• In regression problem, imbalance occurs when there are outlier values that are either much lower or higher than the median or average of the data.

  • For example, while predicting customers lifetime value, most customers may have average spending, but a few high-value customers could skew the dataset.

Note: In many real-world scenarios, the data is inherently imbalanced, such as fraud detection, diagnosis of rare diseases, anomaly detection, etc.

Why is Imbalanced Data a Problem?

• Biased Model Performance:

  • The models trained on imbalanced data tend to be biased towards the majority class, resulting in poor generalization.

• Misleading Evaluation Metrics:

  • Traditional metrics like accuracy can be misleading. A model may achieve high accuracy simply by predicting the majority class most of the time, while completely ignoring the minority class.

How to Deal With Imbalanced Data?

Previously, in this Github repository, I covered three different methods, in detail:

• Data-level methods (Resampling techniques),

• Model-level methods (Algorithm adjustments),

• Metrics-level methods (Choosing better evaluation metrics)

But, is there a better approach? Let’s explore some additional techniques.

How to Fix Imbalanced Data?

Other than resampling the dataset or selecting appropriate algorithms (like tree-based models), you can also consider combining multiple techniques together for better results:

1. Under-Sampling with Ensemble Learning

The idea is to use all samples of the minority class while creating smaller subsets from the majority class, then ensemble multiple models trained on these subsets.

Imagine, you have two classes for binary classification task.

Class A has 1000 examples, while Class B has only 100 examples.

You can break class A into 10 smaller groups, each containing 100 examples.

Train 10 different models, each with one small group from class A plus all examples in class B.

Finally, combine predictions from these models (Ensemble Learning) to improve metrics.

2. Up-Sampling with Adjusted Loss Function

Another method is to adjust the loss function of the model after resampling.

You can up-sample the minority class until a desired ratio is reached and then recalculate the new weights for both classes.

Next, you pass these adjusted weights to the model’s loss function, ensuring that it pays more attention to the minority class during training.

Certainly, these are two of many possible ways to handle class imbalance problems, which are commonly discussed in Data Science interviews.

However, you can experiement with different combinations of techniques. The general idea is to use multiple techniques together to address class imbalance.

Choosing the Right Metrics

Also, picking the right evaluation metric is really crucial when working with imbalanced data sets.

Key Considerations:

Before we talk about which metrics are appropriate for imbalanced data sets there’re two important things to keep in mind:

• First, always evaluate on the original, unmodified dataset rather than resampled data to avoid overfitting.

• Secondly, the test dataset should represent the original data distribution as closely as possible.

As I mentioned earlier, accuracy can be misleading when classes are imbalanced, as the performance of the model on the majority class will dominant the accuracy score.

So, a better choice is to consider using accuracy for each class individually, meaning you should evaluate the model’s performance on the minority class separately to get a clearer picture.

Other metrics, such as the Precision-Recall (PR) curve helps identify a threshold that works best for the dataset, as it gives more emphasis on how many predictions the model got right out of the total number it predicted to be positive, which is helpful when dealing with imbalanced data sets.

Another commonly used metric is the ROC-AUC. From the ROC curve, we can tune thresholds to increase recall while decreasing the false positive rate.

However, the problem with ROC curve is that it treats both classes equally and is less sensitive to improvements in the minority class.

This makes it less helpful compared to the PR curve.

Alright, that’s a wrap! If you’ve made it this far — thank you! Stay tuned with ME, so you won’t miss out on future updates.

Do not forget to explore the full implementation, including code and data: Github Repository 👈🏻

Until next time, happy learning!

— Nikita Prasad

However, after extensive testing, many users, including AI researcher debating on Reddit — suggest that: “GPT-4.5 hallucination rate, in practice, is too high for reasonable use”.

In this read, let’s dive into Hallucinations in LLMs, explore why they happen, and discuss strategies to mitigate them.

Quick Pause : If you find these tips helpful, do not forget to subscribe me and stick around to dive deeper into Python & ML insights. 👇🏻

Subscribe now

What Are LLM Hallucinations?

While LLMs can generate fluent and coherent text on wide range of topics and domains, they’re also prone to hallucinate.

Hallucinations in LLMs refers to the phenomenon where the model deviate from factual accuracy or contextual logic.

This can range from minor inconsistencies to generating entirely fabricated outputs, outright incorrect facts, or erroneous explanations.

You can categorize hallucination across different levels of granularity:

• Sentence Contradictions: The LLM generates statements that contradicts each other within the same response.

  • Something like "Einstein was born in 1879 in Germany. However, his birthplace is Switzerland".

• Prompt Contradictions: The generated response contradicts with the prompt that was used to generate it.

  • Say, if you ask an LLM to write a formal email declining an invitation and it returns a rude response, that would be in direct contradiction to what you asked.

• Factual Contradictions: These are the factual error hallucinations responded by the LLM.

  • For instance, in one test, GPT‑4.5 incorrectly stated that “The Third Circuit held personal jurisdiction existed”. However, the actual ruling was the exact opposite. Such factual errors are especially concerning.

Now with the question of what LLMs hallucinations are answered, let’s understand the reason behind that:

Why Do Hallucinations Happen?

Understanding why LLMs hallucinate is not straightforward, as the way they derive their output remain partly a black box — even to their engineers.

But there’re a number of common causes have been identified:

• Limited Contextual Understanding: Even though models like GPT‑4.5 are pretrained on extensive and diverse datasets, they may still lose vital context when sentences or clauses span long distances.

• Training Data Limitations: If the training data is biased or lacks the necessary knowledge for specific queries, the model might fill in the gaps with “most probable” text rather than the “true” information.

• Output Generation Methods: Newer models might focus more on generating “emotionally intelligent” or so called human-like responses rather than factually verified outputs, leads to potentially convincing yet incorrect responses.

• Benchmarking vs. Real Use: Although OpenAI’s benchmarks (such as the SimpleQA benchmark) claim a reduced hallucination rate of 37.1%, but in practice, these metrics often fail to capture the full complexity of real-world scenarios.

How Can We Fix the Hallucination Problem?

While no solution is perfect, some promising approaches can help reduce hallucinations when using LLMs:

1. Clear and Specific Prompts: The more precise and the more detailed your input prompt, the more likely the LLM will generate relevant and accurate outputs.

2. Active Mitigation Strategies: Adjusting model settings — such as temperature parameter — can control the randomness of the output, and potentially reduce hallucinations.

3. Integrate Web Search: Models that combine reasoning with real-time web search tend to verify facts more accurately, leading to more reliable outputs.

So while LLMs like GPT 4.5 may sometimes hallucinate, understanding the causes and employing the mitigation strategies can help you harness their true potential.

🫵🏻Over to You: What’s your experience with AI hallucinations? Drop a comment below, and let’s discuss! 🖋️👇

Until next time, happy learning!

— Nikita Prasad

You may tweak hyperparameters, increase training data, and even adjust the learning rate—yet the issue persists.

What if the real problem lies in your activation function?

In this guide, let’s break down the most commonly used activation functions, their strengths and weaknesses, and how to choose the right one for your deep learning model.

⏸️ Quick Pause: If you’re new here, I’d highly appreciate if you subscribe to recieve bi-weekly data tips and insights — directly into your inbox. 👇🏻

Subscribe now

➡️ There are two types of Activation Functions:

1. Linear Activation Functions
2. Non-linear Activation Functions

First thing first, let’s answer!

🛠️ Why Use Non-Linear Activation Functions?

In real-life data, we work with non-linear data samples. Adding an activation function ensures that we consider the non-linearity in the data, further helping the DL models to perform well in complex tasks.

Adding non-linearity ensures that:
✔ The model can learn hierarchical representations
✔ It enables deep networks to approximate any function
✔ It helps solve complex real-world problems in NLP, vision, and more

Here's the list of Non-Linear Activation Functions:👇🏻

1️⃣ 𝗦𝗶𝗴𝗺𝗼𝗶𝗱 - It's historically popular function that squashes its input into the range [0, 1]. But it can cause Vanishing Gradient Problems in deep networks.

2️⃣ 𝗧𝗮𝗻𝗵 - Similar to the sigmoid but scales the output to be in the range [-1, 1]. It's zero-centered, which can help mitigate the Vanishing Gradient Problems to some extent.

3️⃣ 𝗥𝗲𝗟𝗨 (Rectified Linear Unit) - It's computationally efficient and helps with the vanishing gradient problem but can cause "Dying ReLU" problem.

4️⃣ 𝗟𝗲𝗮𝗸𝘆 𝗥𝗲𝗟𝘂 - To tackle the dying ReLU problem, Leaky ReLU has a small slope for negative values instead of 0.

5️⃣ 𝗣𝗥𝗲𝗟𝗨 (Parametric ReLU) - Similar to Leaky ReLU but the slope for negative values is learned during training rather than being predefined.

6️⃣ 𝗘𝗟𝗨 (Exponential Linear Unit) - Tries to make the mean activations closer to zero. Transforms negative inputs to values between -α and 0, producing a more robust model.

7️⃣ 𝗦𝗘𝗟𝗨 (Scaled Exponential Linear Unit) - Like ELU, but with scaling, making it self-normalizing. It has specific conditions under which it can maintain mean 0 and variance 1.

8️⃣ 𝗦𝗼𝗳𝘁𝗽𝗹𝘂𝘀 - A smooth approximation to the ReLU function, and it's always positive.

9️⃣ 𝗦𝗼𝗳𝘁𝘀𝗶𝗴𝗻 - Scales input by 1 plus the absolute value. Similar to tanh but less common used.

1️⃣0️⃣ 𝗛𝗮𝗿𝗱 𝗦𝗶𝗴𝗺𝗼𝗶𝗱 - Piecewise linear approximation of sigmoid function, more computationally efficient than the regular sigmoid.

1️⃣1️⃣ 𝗦𝘄𝗶𝘀𝗵 - Self-gated activation function discovered by researchers at Google. It's computationally efficient and outperforms ReLU in some cases.

1️⃣2️⃣ 𝗠𝗶𝘀𝗵 - Combines softplus and tanh, shown to outperform many traditional activations in deep networks.

Note: All these functions are used in hidden layers, while SoftMax is used for Multi-class classification, typically preferred as an output layer function.

I hope this guide helped you gain a deeper understanding of activation functions and their impact on building robust neural network models.

For most business problems, I keep it simple—using ReLU for hidden layers and Softmax for multi-class classification. If the results are reasonable, I move forward. Only after evaluating the final pipeline with the client do I experiment with new layers and activation function combinations to optimize performance.

Comment Down 👇: Which activation function do you use often when training a neural network?

Until next time, happy learning!

— Nikita Prasad

This motivates me to write a detailed yet crucial article on the fundamentals of Attention Mechanism — the core of GPT Models. Along with an implementation of a Simple Attention Mechanism from scratch to truly understand how it works.

⏸️ Quick Pause: If you’re new here, I’d highly appreciate if you subscribe to recieve bi-weekly data tips and insights — directly into your inbox. 👇🏻

Subscribe now

The Problem with Modeling Long Sequences

In tasks like machine translation, word-by-word translation doesn’t work because it requires contextual understanding and grammatical alignment between the source and target languages.

Prior to the introduction of transformer models, encoder-decoder RNNs were commonly used for machine translation tasks.

In this setup, the encoder processes a sequence of tokens from the source language, using a hidden state—a kind of intermediate layer within the neural network.

Leading to loss-of-context, especially in long complex sentences where dependencies might span long distances.

As the current hidden state is condensed representation of entire input sequence into single hidden state vector.

Solution? Self-Attention Mechanism!

What is the Self-Attention Mechanism?

Through an attention mechanism, the text-generating decoder segment of the network is capable of selectively accessing to different parts of the input tokens.

💡 Key Idea: Certain input tokens hold more significance (weight) than others in the generation of a specific output token, to improve LLM performance.

Self-attention in transformers — sometimes referred to as intra-attention — is a mechanism that allows the inputs to interact with each other (“self”) in order to determine what they should focus on (“attention”).

In simple terms, they process n inputs and return n outputs. The outputs comprise the aggregates of these interactions and also attention scores that are calculated based on a single input.

Implementing Simple Attention Mechanism

For illustration purposes, let’s implement a simple version of self-attention, which does not contain any trainable weights (for now).

Suppose we are given an input sequence x₁ to x_T :

• The input is a text (for example, a sentence like "Attention Mechanism drives contextual embedding") that has already been converted into token embeddings.

• For instance, x₁ is a d-dimensional vector representing the word "Attention", x₂ for “Mechanism”, and so forth.

Goal: To compute context vectors, z_i for each input sequence element x_i in x₁ to x_T, where z and x have the same dimension.

The code below walks through the figure above step by step:

In the case of the tensor shown above, each row represents a word, and each column represents an embedding dimension:

import torch

input_emb = torch.tensor([
    [0.12, 0.45, 0.67],  # "Attention"
    [0.34, 0.56, 0.78],  # "Mechanism"
    [0.23, 0.57, 0.91],  # "drives"
    [0.76, 0.88, 0.45],  # "contextual"
    [0.54, 0.12, 0.34]   # "embedding"
], dtype=torch.float32)

We use input sequence element 1, x₁, as an example to compute context vector z₁; later, we will generalize this to compute all context vectors.

Step 1: Attention Scores (ω)

The first step is to compute the unnormalized attention scores by computing the dot product between the query x₁ and all other input tokens:

Raw, unnormalized values that indicates how relevant one input element is to another.

Computed by comparing the query vector (Q) of one element with the key vector (K) of all element.

query = input_emb[0]  # 1st input token is the query

attn_scores_1 = torch.empty(input_emb.shape[0])
for i, x_i in enumerate(input_emb):
    attn_scores_1[i] = torch.dot(x_i, query) # dot product (transpose not necessary here since they are 1-dim vectors)

print(attn_scores_1)

# Output: tensor([0.6658, 0.8154, 0.8938, 0.7887, 0.3466])

Note: A dot product is used for multiplying two vectors elements-wise and summing the resulting products.

Step 2: Attention Weights (α)

It represent the relative importance of one element to another in a probabilistic manner (value between 0 to 1).

Note: Larger weights means greater relevance.

Let’s normalize the unnormalized attention scores ("omegas", ω) so that they sum up to 1:

attn_weights_1_temp = attn_scores_1 / attn_scores_1.sum()

print("Attention weights:", attn_weights_1_temp)
print("Sum:", attn_weights_1_temp.sum())

# Output: Attention weights: tensor([0.1897, 0.2323, 0.2546, 0.2247, 0.0987])
#         Sum: tensor(1.0000)

Note: It is recommended, to use the softmax function for normalization, which is better at handling extreme values and has more desirable gradient properties during training.

So, let’s use the PyTorch implementation of softmax for scaling, which also normalizes the vector elements such that they sum up to 1:

attn_weights_1 = torch.softmax(attn_scores_1, dim=0)

print("Attention weights:", attn_weights_1)
print("Sum:", attn_weights_1.sum())

# Output: Attention weights: tensor([0.1896, 0.2202, 0.2381, 0.2144, 0.1378])
#         Sum: tensor(1.0000)

Step 3: Context Vectors (z)

The input embedding vectors are converted to the context vector.

It aims to capture both semantic and syntactic information from the input embeddings.

It is the key component that encodes the weighted representation of the input sequence, to capture most relevant information for each element in the sequence by considering its relationship with all other tokens.

Note: Context size is the maximum number of previous tokens the LLM looks at before predicting next token.

Let’s, compute the context vector z₁ by multiplying the embedded input tokens, x_i with the attention weights and sum the resulting vectors:

query = input_emb[0] # 1st input token is the query

context_vec_1 = torch.zeros(query.shape)
for i,x_i in enumerate(input_emb):
    context_vec_1 += attn_weights_1[i]*x_i

print(context_vec_1)

# Output: tensor([0.3897, 0.5495, 0.6587])

The model now has a weighted understanding of the input sequence.

Computing Attention Weights for All Input Tokens

Above, we computed the attention weights and context vector for input 1.

Next, let’s generalizing this computation for all tokens in the input embeddings.

Applying step 1 to all pairwise elements to compute the unnormalized attention score matrix:

attn_scores = torch.empty(5, 3)

attn_scores = inputs @ inputs.T # Compute scores for all pairs
print(attn_scores)

"""
Output: 
tensor([[0.6658, 0.8154, 0.8938, 0.7887, 0.3466],
        [0.8154, 1.0376, 1.1072, 1.1022, 0.5160],
        [0.8938, 1.1072, 1.2059, 1.0859, 0.5020],
        [0.7887, 1.1022, 1.0859, 1.5545, 0.6690],
        [0.3466, 0.5160, 0.5020, 0.6690, 0.4216]])
"""

Similar to step 2 previously, we normalize each row so that the values in each row sum to 1:

attn_weights = torch.softmax(attn_scores, dim=-1) # Normalize Attention Scores
print(attn_weights)

"""
Output: 
tensor([[0.1896, 0.2202, 0.2381, 0.2144, 0.1378],
        [0.1766, 0.2206, 0.2365, 0.2353, 0.1309],
        [0.1821, 0.2254, 0.2488, 0.2207, 0.1231],
        [0.1481, 0.2026, 0.1994, 0.3185, 0.1314],
        [0.1721, 0.2039, 0.2010, 0.2376, 0.1855]])
"""

Lastly, applying step 3 to compute all context vectors:

all_context_vecs = attn_weights @ input_emb # Compute all context vectors
print(all_context_vecs)

"""
Output: 
tensor([[0.3897, 0.5495, 0.6587],
        [0.4001, 0.5606, 0.6560],
        [0.3899, 0.5589, 0.6653],
        [0.4455, 0.5898, 0.6267],
        [0.4169, 0.5375, 0.6272]])
"""

As a sanity check, the previously computed context vector z₁ can be found in the 1st row in above:

print("Previous 1st context vector:", context_vec_1)

# Output: Previous 1st context vector: tensor([0.3897, 0.5495, 0.6587])

Now, each token has a dynamically computed representation based on its relationship with all other tokens.

Want to go deeper? Subscribe for upcoming deep dives into Transformer Architectures!

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

And that’s a wrap! If you’ve made it this far — thank you so much, stay tuned with ME, so you won’t miss out on future updates..

Until next time, happy learning!

— Nikita Prasad

How?

👇🏻Consider the following snippet of code:

At first glance, you might expect b to remain [1, 3, 4], but instead, both a and b change. Why?

If you’re a data scientist, ignoring such risks may lead to incorrect analysis, unexpected bugs, and memory issues in ML models.

Quick Pause : If you find these tips helpful, do not forget to subscribe me and stick around to dive deeper into Python & ML insights. 👇🏻

Subscribe now

Let’s break down the issue,

What’s Happening?

When you execute b=a, you aren’t creating a new list.

Instead, both a and b now refer to the same list in the memory.

So, when you do a.append(2), you’re modifying the original list, which means both a and b reflect that changes.

Imagine this type of unintented modification corrupting your entire data science project while working with large datasets.

How to FIX It?

1. Copying a List Using .copy() Method

The .copy() method return a shallow copy of original list.

This works fine as long as the list does not contain nested objects. For instance:

a = [1, 2, 3]  
b = a.copy()  # Now b is a separate list  
a.append(6)  

print(a)     # Output: [1, 2, 3, 6]  
print(b)     # Output: [1, 2, 3]  (Fixed!)  

print(id(a)) # Output: 131938375649664
print(id(b)) # Output: 131938374499648

Now, b stays unchanged because .copy() creates a new list instead of a reference.

However, if the list contains nested objects, .copy() only copies the reference to those nested objects, not the objects themselves.

Below code snippet can help you understand this better:

original_list = [1, [2, 3], [4, 5]]
shallow_copied_list = original_list.copy()

# Modify the original list
original_list[1][0] = 'X'

# Modify the copied list
shallow_copied_list[2][1] = 9

print(original_list)             # Output: [1, ['X', 3], [4, 9]]
print(shallow_copied_list)       # Output: [1, ['X', 3], [4, 9]]

print(id(original_list))         # Output: 133878327171712
print(id(shallow_copied_list))   # Output: 133878327557248

⚠️ Problem: A shallow copy creates a new outer list at a different memory location (hence, a different id).

However, the nested objects inside the list still reference the same memory addresses as in the original list.

So, modifying a nested object will affect both copies.

2. Copying a List Using copy.deepcopy()Function

The copy.deepcopy() creates a completely independent copy, ensuring that all nested objects are also duplicated rather tha just their references.

import copy

original_list = [1, [2, 3], [4, 5]]
shallow_copied_list = copy.copy(original_list)
deep_copied_list = copy.deepcopy(original_list) 

# Modify the original list
original_list[1][0] = 'X'

print(original_list)            # Output: [1, ['X', 3], [4, 5]]
print(shallow_copied_list)      # Output: [1, ['X', 3], [4, 5]]
print(deep_copied_list)         # Output: [1, [2, 3], [4, 5]] (No unintended modification)

print(id(original_list))         # Output: 140288711654720
print(id(shallow_copied_list))   # Output: 140288711602432
print(id(deep_copied_list))      # Output: 140288711597440

Now, the deep-copied list remains completely independent because deepcopy() recursively copies all nested objects.

3. Avoid Using Lists as Default Arguments in Functions

Using mutual objects like lists as default function arguments can create unexpected behavior, that you can read here.

Additional, if as a curious Python developer, you also wonder:

• How is data processed and manipulated in memory?

• How has it affected the quality of the program?

This article will provide a Comprehensive overview of Mutable vs Immutable Objects in Python and why they are crucial for effective programming.

Stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad

And looking for, what can you do to make the process easier.

Stay with me as I walk you through some useful tips to break down your analysis into digestible chunks.

⏸️ Quick Pause: If you’re new here, I’d highly appreciate if you subscribe to recieve bi-weekly data tips and insights — directly into your inbox. 👇🏻

Subscribe now

So, let’s begin!

Data can be complex and difficult to understand.

As a data analyst or scientist, you can make data more relatable and easier for your audience—particularly non-technical stakeholders in the corporate world—by weaving it into a story.

For the reason data isn’t just numbers — it’s the foundation for better decision-making across all domains, from product development to marketing and financial forecasting.

How to Present Your Data Effectively?

As a data analyst, your role is to make sense out of past and present data to uncover trends and relationships of your key metrics to report observation or insights about what has happened to take informed business decisions.

Use these simple rules to to standout in your presentations:

1. Even if it seems boring, begin by explaining WHY the analysis is being conducted. This ensures everyone is on the same page.

2. Be brief with the HOW to explain how you arrived at the results — usually Your Methodology.

3. Then move to the WHAT, present the results, insights, and recommendations. Offer a side-by-side comparison of different KPIs and Metrics. Visuals can help to make your story more engaging and memorable.

4. Lastly, always highlight the key takeaways for your audience, particularly regarding future needs. So that your audience understands their roles, and takes action accordingly.

Here’s the key:

Tell a clear and concise story. Don't overload your colleagues with too much information.

➡️ Also, do not miss: Choosing the Right Charts for Effective Business Insights

Thanks for reading!

Until next time, happy learning!

— Nikita Prasad

Whether you’re forecasting sales, setting optimal product prices, or predicting customer behavior, understanding regression techniques is essential.

So, let’s dive into six key regression techniques you should know and when to use them.

Subscribe as here, I want to make Machine Learning easy for you. 👇🏻

Subscribe now

1. Linear Regression — The Classic Approch

If you’re new to regression, Linear Regression is where it all begins.

It’s simple, easy to interpret and works well when your data follows a linear relationship (a stright-line trend).

Here’s the Github Repository (⭐ this repo):

• Simple Linear Regression

• Multiple Linear Regression

Use it when:

• You want to predict a continuous outcome based on one or more features.

• Your data has a roughly straight-line relationship.

2. Polynomial Regression — Non-Linear Relationships

Not all data follows a straight-line pattern — sometimes, it curves!

That’s where you need Polynomial Regression — it fits a curved trend by adding higher degree terms to the equation instead of fitting a straight line.

Here’s the Github Repository (⭐ this repo):

• Polynomial Regression

Use it when:

• Your data shows a non-linear trend.

• You need to capture curvatures in the relationship.

3. Ridge Regression — Handling Overfitting

When your model learns too much from the training data and struggles to generalize, Ridge Regression helps by adding a penalty term to shrink coefficients.

This penalty term make the model less complex and results in better predictions.

Here’s the Github Repository (⭐ this repo):

• Ridge Regression (L2 Regularization)

• Ridge Regression Using Gradient Descent

Use it when:

• You’re working with high-dimensional data.

• Overfitting is a concern in your model.

4. Lasso Regression — Feature Selection Included

Lasso Regression is similar to Ridge but goes one step further — it can shrink some coefficients to exactly zero.

This is helpful in automatically selecting the most important features while dropping the rest.

Here’s the Github Repository (⭐ this repo):

• LASSO Regression (L1 Regularization)

Use it when:

• You want both regularization and feature selection.

• Your dataset has a lot of irrelevant or redundant variables.

5. Elastic Net — The Best of Both Worlds

Can’t decide between Ridge and Lasso?

Elastic Net combines both!

It strikes a balance by applying both L1 (Lasso) and L2 (Ridge) regularization, making it a great choice when you suspect some features should be eliminated but not all.

Use it when:

• Your dataset is high-dimensional and may have correlated features.

• You need better stability than Lasso alone.

6. Logistic Regression — Classification problems

Despite its name, Logistic Regression isn’t used for predicting numbers—it’s for classification (yes/no, true/false, spam/not spam, etc).

Instead of predicting a continuous value, it predicts the probability of belonging to a category.

Here’s the Github Repository (⭐ this repo):

• Logistic Regression

Use it when:

• Your target variable is binary (e.g., pass/fail, buy/don’t buy).

• You need an interpretable model for decision-making.

Wrapping Up – Which Regression Model Should You Use?

There’s no one-size-fits-all regression technique—it all depends on your data and problem!

Next time you're working on a machine learning project, take a step back and think about which regression method fits best.

Picking the right one could make all the difference in reliability of your predictions!

If you’d like to explore the full implementation, including code and data, then do not forget to checkout the given repositories.

👉 Over to you:

Which regression technique do you use the most?

Drop your thoughts in the comments!

Thanks for reading!

Until next time, happy learning!

— Nikita Prasad

Let’s break this down in simple terms:

Quick Pause: If you’re new here Subscribe — my goal is to make Data Science easy for you. 👇🏻

Subscribe now

What are Outliers? 🤔

An outlier is an observation in a dataset that lies far from the rest of the values.

Or simply stating, data points that are significantly different from the other values in a dataset.

That means an outlier can be much larger or smaller than the majority of values, making them stand out.

Why Do they Occur?

An outlier may occur due to the variability in the data, or due to experimental bias or human error.

Note: They may indicate heavy skewness in the data (heavy-tailed distribution).

What Do They Affect?

Outliers can distort the statistical results, leading to misleading conclusions.

Their impact depends on which statistical measure you’re using.

• Mean is the only measure of central tendency that is affected by the outliers which in turn impacts Standard Deviation.

• Median is commonly preferred, when dealing with skewed data.

• Mode is used if there are outliers and about ½ or more of the data is the same.

How to Detect Outliers?

If datasets are small, you can often spot outlier by just looking at the data.

But for larger datasets, we need proper visualization and statistical techniques to accurately find them.

Below are some of the techniques of detecting outliers:

1. Z-scores (Standard Deviation Method) :

• Criteria: Works best for normally distributed data.

• Rule: Any data point whose Z-score falls out of ±3rd standard deviation is considered as an outlier.

• Formula: Z = X−μ​ / σ

  where X is the data point, μ is the mean, and σ is the standard deviation.

2. Inter Quantile Range(IQR) Method

• Criteria: Best for skewed or non-normally distributed data points.

• Rule: Any data that lie 1.5 times of IQR above Q3 and below Q1 are outliers.

In statistics, interquartile range or IQR is a quantity that measures the difference between the first and the third quartiles in a given dataset.

• Formula: IQR = Q3 − Q1

• Outlier Thresholds: Lower Bound = Q1 - 1.5 × IQR

  Upper Bound = Q3 + 1.5 × IQR

Note : Boxplot highlights outliers as individual points that fall outside the “whiskers” of the box.

3. Detecting Outliers Using Percentile

Defining a custom range that accommodates all data points that lie anywhere between 0.5 and 99.5 percentile of the dataset.

How to Handle Outliers?

Once you’ve identified outliers, the next question is: What should you do with them?

The answer depends on the cause of the outlier and its impact on your analysis.

Below are some of the methods of treating the outliers:

Method 1: Trimming/Removing Outliers

If an outlier is caused by human error or faulty measurement, it’s safe to remove it.

Although, it isn’t a good practice to follow this, without a valid reason, as it can distort the real data, reducing its accuracy.

Method 2: Quantile Based Flooring and Capping (Winsorization)

Instead of removing outliers, you can limit their impact by capping at a certain value, say above the 90th percentile value or floored at a factor below the 10th percentile value.

Method 3: Median Imputation

As the mean value is highly influenced by the outliers, it is advised to replace the outliers with the median value, to preserve dataset’s integrity.

Method 4: Use Transformations

It is recommended to apply transformations when dealing with highly skewed data.

Logarithmic or Square Root Transformations can reduce the influence of extreme values.

When to Keep Outliers?

YES, not all outliers should be removed!

Sometimes, they provide important insights, that is why we need to keep them.

• In Fraud Detection, extreme values could indicate fraudulent transactions.

• In Medical Data, rare values might hightlight a significant medical condition.

• In Business Analysis, an unexpected spike in sales might indicate a successful marketing campaign.

So, always investigate the cause before deciding to remove or modify outliers.

If you’d like to explore the full implementation, including code and data, then checkout: Github Repository 👈🏻

If you enjoyed this deep dive, stay tuned with ME, so you won’t miss out on future updates.

Until next time, happy learning!

— Nikita Prasad