Featured

Image Processing & Background Removal: Beyond reblog

Andrew Smalley

22 Feb 2025 • 7 min read

Introduction

In image editing, automatically removing the background is no longer a novelty—it's practically expected. From e-commerce product shots to social media posts, clean and precise background removal can significantly enhance an image's presentation. One well-known command-line tool that provides this functionality is rembg, which uses advanced neural network models to separate foreground from background. But how exactly does this technology work, and can we build our background removal app using the same models but without relying on rembg directly?

In this post, we'll cover:

How background removal models work and why they're so effective.
Where to find open-source models for image segmentation.
rembg and other popular background removal tools.
Building a simple "rembg-like" app using the same deep learning model.

1. How Background Removal Models Work

A Quick Overview of Image Segmentation

At the core of background removal is image segmentation, where a model classifies each pixel in an image as either foreground or background. Deep learning often achieves this with convolutional neural networks (CNNs) trained on thousands—or even millions—of annotated images. The neural network learns to identify the edges and shapes that typically define a subject, such as a person, product, or object, and separate it from its background.

U^2-Net and Other Architectures

One commonly used model is U^2-Net, an open-source architecture known for its accuracy in saliency detection (identifying the most "important" region in an image). Tools like rembg typically rely on such networks. The "U" shape in these architectures refers to a symmetrical encoder-decoder design, which processes the image at different scales to capture fine details (like hair or curved edges) and broader shapes.

Why It Works:

Multi-scale Analysis: The network analyses the image at various resolutions, learning both macro-level shapes and micro-level details.
Skip Connections: Information from earlier layers (high-resolution details) is carried over to later layers, ensuring that fine edges remain sharp.
Massive Training Sets: These models are often trained on large datasets containing diverse backgrounds and objects, making them robust to various conditions (e.g., clutter, unusual lighting).

Why It's So Effective

Deep learning segmentation models don't rely on simplistic rules or chroma-keying (like green screens). Instead, they truly "understand" edges and shapes, allowing them to handle more complex backgrounds. The result is a more accurate separation that requires minimal manual touch-up.

2. Where to Find Open-Source Models

If you'd like to explore these models directly, there are several places to look:

GitHub Repositories: Projects such as xuebinqin/U-2-Net provide the original code and pre-trained weights for U^2-Net.
Other models from https://github.com/danielgatis/rembg Github
u2net (download, source): A pre-trained model for general use cases.
u2netp (download, source): A lightweight version of u2net model.
u2net_human_seg (download, source): A pre-trained model for human segmentation.
u2net_cloth_seg (download, source): A pre-trained model for Cloths Parsing from human portrait. Here clothes are parsed into 3 category: Upper body, Lower body and Full body.
silueta (download, source): Same as u2net but the size is reduced to 43Mb.
isnet-general-use (download, source): A new pre-trained model for general use cases.
isnet-anime (download, source): A high-accuracy segmentation for anime character.
sam (download encoder, download decoder, source): A pre-trained model for any use cases.
birefnet-general (download, source): A pre-trained model for general use cases.
birefnet-general-lite (download, source): A light pre-trained model for general use cases.
birefnet-portrait (download, source): A pre-trained model for human portraits.
birefnet-dis (download, source): A pre-trained model for dichotomous image segmentation (DIS).
birefnet-hrsod (download, source): A pre-trained model for high-resolution salient object detection (HRSOD).
birefnet-cod (download, source): A pre-trained model for concealed object detection (COD).
birefnet-massive (download, source): A pre-trained model with massive dataset.
Model Zoos: Frameworks like PyTorch and TensorFlow have "model zoos" or hubs where you can download pre-trained segmentation models.
Hugging Face Model Hub: A popular hub for hosting and discovering pre-trained models, often with interactive demos and user-contributed fine-tunings.

Tip: Ensure that you check the licensing. Most open-source segmentation models are available under permissive licences but always read the terms if you plan to integrate them into a commercial product.

3. rembg and Other Popular Tools

rembg

What It Is: A command-line tool (and Python library) that uses U^2-Net to remove backgrounds from images.
Key Features: Simple CLI usage, decent default performance, and minimal configuration required.
Why People Love It: It's free, open-source, and typically yields high-quality results with no fuss.

remove.bg

What It Is: A commercial web-based service for background removal.
Key Features: Offers an API, fast processing, and a simple user interface.
Why People Love It: Quick, user-friendly, and can be integrated into websites or applications. However, it's a paid service with usage limits.

Adobe Photoshop's "Select Subject"

What It Is: A built-in feature in Adobe Photoshop that uses AI to select the main subject in an image.
Key Features: Part of the Photoshop ecosystem, easy to refine the selection manually.
Why People Love It: Seamless integration with other Photoshop features, but requires an Adobe subscription.

PhotoRoom

What It Is: A mobile and web application focusing on product photos.
Key Features: Offers background removal, object editing, and design templates.
Why People Love It: Quick, intuitive, and especially useful for small businesses that need consistent product shots.

Conclusion on Tools: Rembg is a substantial choice of simplicity and open-source reliability. However, a commercial solution might be more suitable if you need advanced design tools or want to outsource the hosting.

4. Building a Simple "rembg-Like" App Using the Same Model

Now, let's build a straightforward background removal app that uses the U^2-Net model (or any similar segmentation model) directly, rather than calling rembg. This approach gives you complete control over custom features, user interfaces, and integrations.

Step 1: Set Up Your Python Environment

mkdir bg-removal-app
cd bg-removal-app
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
pip install torch torchvision pillow opencv-python numpy

Step 2: Download the U^2-Net Model

wget https://github.com/xuebinqin/U-2-Net/releases/download/v1.0/u2net.pth
# Alternatively, store the .pth file in your local directory

Step 3: Write the Core Script

# main.py
import torch
import torchvision.transforms as T
from PIL import Image
import numpy as np
import cv2

from u2net import U2NET  # Example: Suppose we have a local file u2net.py with the model class

def load_model(model_path="u2net.pth"):
    model = U2NET()  # This class definition is in a local 'u2net.py'
    model.load_state_dict(torch.load(model_path, map_location=torch.device('cpu')))
    model.eval()
    return model

def remove_background(input_image_path, output_image_path, model):
    # 1. Load and preprocess image
    img = Image.open(input_image_path).convert("RGB")
    transform = T.Compose([
        T.Resize((320, 320)),
        T.ToTensor()
    ])
    img_tensor = transform(img).unsqueeze(0)  # shape: (1, 3, 320, 320)
    
    # 2. Run inference
    with torch.no_grad():
        pred = model(img_tensor)[0]  # Suppose the model returns a single-channel mask
    pred_mask = pred.squeeze().numpy()
    
    # 3. Post-process the mask
    # Normalise mask to 0-255
    pred_mask = (pred_mask - pred_mask.min()) / (pred_mask.max() - pred_mask.min())
    pred_mask = (pred_mask * 255).astype(np.uint8)
    
    # 4. Resize mask back to original size
    orig_w, orig_h = img.size
    pred_mask = cv2.resize(pred_mask, (orig_w, orig_h), interpolation=cv2.INTER_AREA)
    
    # 5. Create a transparent background
    img_rgba = img.convert("RGBA")
    datas = img_rgba.getdata()
    
    newData = []
    for i, pixel in enumerate(datas):
        alpha = 255 - pred_mask[i]  # invert if the model is a 'foreground' mask
        newData.append((pixel[0], pixel[1], pixel[2], alpha))
    
    img_rgba.putdata(newData)
    img_rgba.save(output_image_path, "PNG")

if __name__ == "__main__":
    model = load_model("u2net.pth")
    remove_background("example_input.jpg", "output.png", model)
    print("Background removal complete!")

Notes:

This script is a simplified version. In reality, you'd want to handle batch processing, advanced mask post-processing, and error handling.
The U2NET class might come from an official PyTorch implementation or a local file referencing the official GitHub code.

Step 4: Add a Minimal Web Interface (Optional)

If you'd like a simple web interface, you can use Flask:

pip install flask

# app.py
from flask import Flask, request, render_template, send_file
import os
from main import load_model, remove_background

app = Flask(__name__)
model = load_model("u2net.pth")

@app.route("/", methods=["GET", "POST"])
def index():
    if request.method == "POST":
        file = request.files["image"]
        input_path = os.path.join("uploads", file.filename)
        file.save(input_path)
        
        output_path = os.path.join("outputs", "out_" + file.filename)
        remove_background(input_path, output_path, model)
        
        return send_file(output_path, mimetype="image/png")
    return '''
    <form method="POST" enctype="multipart/form-data">
      <input type="file" name="image" />
      <input type="submit" />
    </form>
    '''

if __name__ == "__main__":
    os.makedirs("uploads", exist_ok=True)
    os.makedirs("outputs", exist_ok=True)
    app.run(debug=True, port=5000)

Usage:

Run python app.py.
Open your browser at http://localhost:5000.
Upload an image and receive a processed PNG with a removed background.

Step 5: Automate with Docker or a CI/CD

If you're distributing your app, consider packaging it in Docker:

# Dockerfile
FROM python:3.9
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Then build and run:

docker build -t bg-removal .
docker run -p 5000:5000 bg-removal

You can push this image to Docker Hub or any container registry and integrate it into your CI/CD pipeline for automated deployment.

5. Why Build Your App?

Customisation: You can fine-tune the model or add unique features like custom filters or style transfers.
Offline Use: Hosting the model locally is beneficial if you don't want to rely on an external service.
Privacy & Control: Handling images in-house is crucial if you have strict data protection requirements.
Cost Efficiency: Commercial services can be expensive at scale. Open-source solutions let you pay only for infrastructure.

6. Recap of Tools & Approaches

rembg: A straightforward command-line tool using U^2-Net.
remove.bg: A commercial option with an easy-to-use web interface.
Adobe Photoshop: Integrates advanced AI background removal but is subscription-based.
Custom Approach: Using the same underlying model (e.g. U^2-Net) for complete control.

Final Thoughts

Thanks to deep learning, background removal has become increasingly sophisticated. Tools like rembg demonstrate how effectively an open-source model can isolate subjects from their surroundings, producing near-professional results. However, if you want more customisation—such as building a unique user interface or embedding advanced transformations—then leveraging the same model under the hood can be a powerful strategy.

You gain flexibility and cost savings by understanding how these neural networks work, where to find them, and how to integrate them into your application. Whether you're an e-commerce entrepreneur looking to automate product photography or a developer wanting to embed image editing into a more extensive system, open-source background removal is a powerful option. With the code snippets provided, you're well on creating your own "rembg-like" solution.

Need More Help?

If you want to customise your pipeline further, fine-tune the model, or incorporate advanced post-processing techniques, feel free to reach out or explore the many open-source communities dedicated to image segmentation. With creativity and knowledge, you can build a robust, tailored background removal solution that stands out.