Infinite-ISP Tutorial: Infinite-ISP as a Sandbox

Today, open-source projects serve as powerful tools, facilitating collaboration among students and developers for code modification and optimization. Our project, Infinite-ISP is one such example. At the core of digital cameras lies the Image Signal Processor (ISP), responsible for converting raw sensor data into the final RGB image. Infinite-ISP offers a user-friendly Python-based ISP implementation. It serves as a hands-on educational resource, guiding users through the complete image processing pipeline. Infinite-ISP allows individuals to explore various camera ISP operations, from addressing sensor-specific issues to denoising, color correction, image scaling, and compression, making it an invaluable platform for algorithm refinement and comprehensive image processing understanding.

In image processing (IP), it is critical to acknowledge that a significant portion of existing techniques primarily operate on processed and compressed RGB (Red, Green, Blue) data, avoiding the use of raw data. While these approaches have clearly yielded impressive results in a variety of applications, they often overlook the potential buried within the unprocessed raw data captured by image sensors. The raw image data, unaltered by in-camera processing, contains a wealth of information that, if utilized effectively, can lead to significant improvements in image processing algorithms. Relying solely on preprocessed RGB data can inadvertently limit the performance of these algorithms. For example denoising a raw image will give better results as compared to denoising RGB image in which the noise has already been spread out from one channel to another while debayering the raw image.

Introduction to Infinite-ISP

Infinite-ISP is a comprehensive collection of a total of 18 camera pipeline modules, designed to convert input RAW image captured from a sensor to a vibrant RGB/YUV image. Each of these modules is configurable through a range of parameters using a configuration file. Some parameters are configured through a dedicated software tuning tool, while others offer more direct configurability. The entire pipeline is divided into four sections, each corresponding to a specific color space as explained below:

  • Raw Data Processing (Modules 1-8): This section works with raw data to correct sensor and lens-specific artifacts such as dead pixels and fixed pattern noise. It also applies operations like digital gain and white balance to adjust exposure and image coloration.
  • RGB Image Processing (Modules 9-11): After demosaicing in Module 8, the raw file is converted into a 3-channel RGB image through interpolation. In Modules 10 and 11, color correction and gamma correction are applied to the RGB image ensuring a precise display.
  • YUV Image Processing (Modules 12-15): The image is then converted to YUV color space, where denoising takes place in Module 12.
  • YUV/RGB Image Processing (Modules 12-15): The subsequent modules work with either YUV or RGB images, depending on the execution of the RGB conversion module. Here, the image can be resized and compressed before being returned as the final result.

The real motivation behind Infinite-ISP is to streamline procedures, improve current approaches, and develop advanced algorithms in the ISP and computational photography field. This project provides a platform for relevant developers to work on the most intriguing and challenging hardware-related issues that an ISP engineer encounters. The “open innovation” approach, enables anyone with the right skills, time, and interest to contribute generously to the Infinite-ISP project.

Infinite-ISP as IP guide

Infinite-ISP can be integrated with various research projects across different fields. We’ll cover topics like computational photography, algorithm development, application specific ISP tuning and image forensics. Here are some instances of how Infinite-ISP can be a valuable tool to boost your grasp on various image processing algorithms.

Computational Photography

An open-source ISP (Image Signal Processor) plays a pivotal role in advancing the field of computational photography. By granting access and control over the image processing pipeline, it allows the computational photography enthusiasts and researchers to innovate in several ways. It allows for the development of customized image processing algorithms to enhance image quality and enables real-time feedback for creative image manipulation. Discussed below are some of the basic IP techniques that can be learnt in combination with Infinite-ISP.

Denoising

In the context of denoising, one can:

  • Learn about different types of image noise (like bayer noise and FPN) and how various modules in a camera ISP remove them from raw images.
  • Understand the difference between noise caused by the camera’s sensor and random noise in images.
Color Grading
  • One can explore the addition of new modules (discussed under algorithm development) that can improve image quality by enhancing contrast, brightness, and tuning colors. These enhancements also ensure that all images and footage captured during a session or project have a uniform color profile, reducing the need for extensive editing after the initial capture.
Impact of Parameter Configuration on Image Quality
  • Open-source ISPs allow individuals to explore the impact of parameter configurations on image quality,  allowing them to distinguish between alternative ways of processing an image and their outcomes.

Algorithm Development

Infinite-ISP, being a versatile platform for processing raw images, not only allows the users to tune/improve the implemented modules but also enables them to become a valuable contributor to our project by suggesting new algorithms to enhance image quality.

Before we dive in, make sure you have Infinite-ISP cloned and configured. You can find installation instructions here. Create a new branch using the following command.

git checkout -b branch_name
Module class structure

All modules are implemented as classes with some common class attributes and methods and can be found in the ./modules directory.

Each module takes the raw image, sensor specifications and module parameters from the config file. All parameters are encapsulated within python dictionaries. Some common class attributes and methods along with their brief description are as below:

Common class attributes among different modules 

Class Attribute name Description Type
Image Raw image from sensor or output of the previous module ndarray with dtype uint16
sensor_info Sensor specifications from config dict
parm_<module_name> Module specific parameters from config file dict
platform contains configuration parameters that are not part of the ISP pipeline but helps in pipeline execution and debugging dict
is_enable Flag to enable/disable module boolean
is_debug Flag to printing detailed logs for debugging purposes boolean
is_save Flag to save module output boolean

Common methods among different modules

Method Name Description Arguments Output
__init__ To initialize class attributes Img, sensor_info, module parameters, platform(for some modules) None
apply_<module_name> Method with algorithm implementation Varies across modules module output ndarray with dtype uint16
save To save module output None None
execute Calls apply method to execute a module None module output array with dtype uint16
Module implementation

To add a new module to Infinite-ISP, first create a Python file in the “./modules” directory. A good place to start is by replicating an existing module file and then altering it as needed. You’ll mainly need to plug your algorithm under the apply method keeping execute and save methods the same. While implementing your module, ensure that module input and output are uint16 ndarrays for seamless integration with infinite-ISP pipeline.

Let’s add the sharpen module in Infinite-ISP as an example.

Copy and paste a template

I’ve copy and pasted the oecf.py file to start with as this implementation is the most similar to what the sharpen implementation is going to be, including a single apply function along with save and execute methods.

Structure of OECF module

Rename the file and start editing

I am making the following changes in the file:

  • Renaming the python file to sharpen.py
  • Renaming the class object and updating the doc strings
  • Update the __init__ method
    • Removing platform dictionary
    • Replacing oecf by sha for sharpening
  • Update the apply method:
    • Implement a sharpening algorithm
    • Adding a helper function to generate a gaussian kernel for convolution to improve readability of the code.
    • Importing respective python libraries (is needed)
    • Updating the filename tag in save method
    • Update execute method
Plugging your module in Infinite-ISP

Position of your module in the pipeline using the context in which your module operates and interacts with various color spaces. The image below summarizes the working of infinite-ISP emphasizing transitions between color spaces.

Here, I want to apply sharpening on a YUV image so I’m adding the sharpen module between the CSC and 2DNR modules.

Add your module block in the default config file (placed at ./config directory) at the respective position. This step is not necessary but helps as a reminder of the pipeline sequence.

Modify the infinite_isp.py file to include the newly implemented module as a part of the infinite-ISP object. Following modifications are required:

  • Go to file ./infinit_isp.py
  • Import your module
  • Modify the load_config method to load your module parameters from config
  • Modify the run_pipeline method:
    • Place your module block in the pipeline
    • Pass the output of the previous module as input to the new module
    • Similarly, pass the output of this new module to the next module in the pipeline.
Module Verification

Now that you have successfully integrated your module into the pipeline, let’s see it in action. Enable your module in the config file and run infinite_isp.py file. You can use the pipeline output or module output (using is_save flag) for visual assessment of the processed image or use some image quality metrics to compare results.

With Sharpening

Without Sharpening

Code Formatting

Once you have resolved all errors and the module is working properly in combination with other modules you can work on the code linting to write clean and readable code. Pylint library is used for uniform code styling across the project. Install Pylint and pylint-runner using the commands below to see your score.

pip install pylint
pip install pylint_runner
pylint_runner

You can also use black library for a quick fix for white spaces etc.

pip install black
black ./modules/<new_module>.py

Refer to the suggested changes by pylint to style your code. You can generate a pull request on github once you have achieved a score 10 indicated by the following msg generated by pylint.

Application specific ISP tuning

When tuning an ISP for a consumer camera, the top priority is ensuring the highest image quality. However, this priority shifts significantly when tuning the ISP for other purposes where images are not intended for human perception. A prime example of this is in autonomous cars, where the camera footage is primarily used by the vehicle’s systems to detect obstacles and navigate safely. In this context, the concept of image quality becomes irrelevant and offers room for developers to make trade-offs between image quality and the performance of the trained network by adjusting ISP parameters accordingly.

The primary goal here is to work with raw image data rather than preprocessed RGB data to maximize the performance of the trained network, particularly for tasks such as object detection and image segmentation. This approach has shown significant promise and has been validated by multiple research studies. By optimizing the ISP parameters for these non-human perception tasks, developers can enhance the overall performance of autonomous vehicles and similar systems.

We have utilized this framework for CV applications through an Auto Camera Tuning Framework. To start with, we have a default set of ISP parameters through which we generate an image. This image is to be utilized by a CV Application, e.g. detection. The detection task would output a certain accuracy that is attained on this image. In order to enhance this accuracy without having to alter the detection model, we would need to modify the image being generated by the ISP. In short, we would have to learn how the detection results could be improved by only changing the ISP parameters. This is done through mathematical optimization. We have observed improved performances in object detection tasks through this framework. 

One intriguing application could be to process raw medical images to enhance critical segmentation tasks such as tumor segmentation in radiogenomics. 

Overview and Empirical Analysis of ISP Parameter Tuning for Visual Perception in Autonomous Driving

Image Forensics

Image forensics is a field that verifies the origin and authenticity of digital images, detecting any potential manipulation or tampering. It ensures the reliability and credibility of images in various contexts, including criminal investigations and journalism.

Open-source ISPs play a crucial role in image forensics by providing transparency into the inner workings of an ISP. They offer control over the image processing pipeline, allowing forensic analysts to carefully inspect digital images. With access to the image processing code, experts can delve into metadata, compression artifacts, noise patterns, and other digital footprints, exposing potential manipulation. This transparency not only aids in identifying tampering but also deepens the understanding of how an ISP functions, thereby enhancing the investigative capabilities of forensic specialists. Open-source ISPs empower image forensics professionals with valuable insights into ISP operations, reinforcing the accuracy and reliability of visual evidence in legal proceedings and critical contexts where image authenticity is pivotal.