ELEC 4727/5727 – Computer Vision Histogram Matching
Background Histograms can be effective tools for fast object recognition (identification) and object classification, as the data collected does not require feature correspondence (detection of specific details or relationships of object structures.) With respect to images, histograms are the collected counts of data organized into a number of predefined bins.
Histogram matching can serve a number of computer vision and search algorithms. For example, Content-Based Image Retrieval (CBIR), shown in the above figure, is an image comparison system allowing the search of digital test image in a large image database. Content based image means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags, and/or descriptions associated with the image. The term ‘content’ in this context might refer to colors, shapes, textures, or any other information that can be derived from the image itself. CBIR is desirable because most image search engines rely purely on metadata and this produces a lot of garbage in the results. OpenCV Functions There are 2 core OpenCV functions to help with this process:
• cv2.calcHist • cv.CompareHist
cv2.calcHist(images, channels, mask, histSize, ranges) → hist Parameters:
images – Source arrays. They all should have the same depth, CV_8U or CV_32F , and the same size. Each of them can have an arbitrary number of channels. channels – List of the dims channels used to compute the histogram. mask – Optional mask. If the matrix is not empty, it must be an 8-bit array of the same size as images[i] . The non-zero mask elements mark the array elements counted in the histogram. histSize – Array of histogram sizes in each dimension. ranges – Array of the dims arrays of the histogram bin boundaries in each dimension. Specify the lower (inclusive) boundary of the histogram bin and the upper (exclusive) boundary. The values for ranges and histSize must be compatible.
The function for comparing histogram: cv2.compareHist(H1, H2, method) → retval Parameters:
H1 – histogram 1 H2 – histogram 2 method – comparison method
cv2.CV_COMP_CORREL Correlation cv2.CV_COMP_CHISQR Chi-Square cv2.CV_COMP_INTERSECT Intersection cv2.CV_COMP_BHATTACHARYYA Bhattacharyya distance
Background Currently the histogram code will be discussed in class and students should follow to setup a Jupyter notebook to take processing the OpenCV images. The goal of this exercise is to perform histogram comparisons and similarity ranking of all of the images. Please review the Jupyter Notebook : CV_Histogram_Practice.ipynb Transfer this to your Linux server account. The guide is prepared for you to practice with histograms.
No image mask
Num Bins Lower (inclusive) to Upper (Exclusive) Bin Values
Gray Scale Histogram Setting
Background – Pokemon Use the following 4 characters, the assignment is to construct a histogram-matching framework to recognize matching images of the same class of pokemon.
DataSet: (/home/CV/HistogramDataset) There are several directories to integrate into your solution. However, a .zip has been posted for you to view the images separately on your own system of even run OpenCV on your own system. targets_directory = ‘targets’ class_directories = [‘bulbasaur’,’charmander’,’pikachu’,’squirtle’] mystery_directory = ‘mystery’ image_directory = ‘/home/CV/HistogramDataset/’ ├── classes │ ├── pikachu [10 entries] │ ├── bulbasaur [10 entries] │ ├── charmander [10 entries] │ └── squirtle [10 entries] ├── targets [4 entries] ├── mystery [10 entries]
[Task] Generate Histograms for Images Build a python framework with modular function that takes the target images and generates a histogram for each target file and compares to each set of scene images (and the test files). Note: The Jupyter Notebook has been provided to help with some understanding of the histograms and how to get either 3 histograms (one per channel) or a single composite histogram. However it is the decision of the developer to explore any design of the histogram (number of channels, number of bins per channel, normalize the values, filter the counts, etc). For example, your system might determine that if a template has rich BLUE values, then selectively dropping or lowering the number of bin counts for the other channels (RED,GREEN) is the key to performing good matching. In this way, the coded system can take advantage of tailoring the algorithm to the 4 base target images. Shown below is an example of plotting the RGB channel data for an image. This may help in first exploring what ideas to perform matches. https://docs.opencv.org/master/d1/db7/tutorial_py_histogram_begins.html
[Task] Compare Histograms between Images While you are constructing a custom system for image recognition, the framework must perform certain evaluations. Build modular Python functions that compares the histogram values from four functions:Correlation, Chi-Square, Intersection, Bhattacharyya distance
The following is a Python tuple that can be used. The three elements give the string name, the OpenCV function, and whether the data values range from low score matching to high score. OPENCV_METHODS = ((“Correlation”, cv2.HISTCMP_CORREL, True), (“Chi-Squared”, cv2.HISTCMP_CHISQR, False), (“Intersection”, cv2.HISTCMP_INTERSECT, True), (“Hellinger”, cv2.HISTCMP_BHATTACHARYYA, False))
[REQUIREMENT] Your code must make use of a dictionary that maps the keys (filenames) to some OpenCV histogram framework of scores. Example: histogram_pikachu would collect each file from the class folder classes/pikachu and have calculated a histogram. Example key ‘1.png’ -> [23, 23 ,200, 30] would be a 4-entry histogram. You must have a function that when presented with a single image (and/or respective histogram), and the histogram of a class like the histogram_pikachu, the result of the function returns a score. To then properly take any mystery image and guess the type, your framework should run the mystery image with some histogram comparison technique across the 4 dictionaries and return one of 5 answers about the image: [‘bulbasaur’,’charmander’,’pikachu’,’squirtle’, ‘no _match’] [TASK] Describing your histogram matching approach Use the histogram score comparisons of each of the 4 OpenCV histogram techniques to rank the scores of each class image from the 4 distinct scene types. Sort the scores and track the values such that the data structures can be accessed easily, plotted for viewing, and used in making decisions. You must describe your final technique and should use graphs/plots to prove your decision. Another example would be that you believe running all 4 histogram techniques and weighing all outputs might be the best way of deciding a match. Show/state which of the comparison techniques are best for each target image. This is open-ended, but there must be a clear description. [TASK] Mystery image classification There are 10 images in the mystery folder. Your framework must run the directory location, read the images and classify each image. The final output should be the list of files names in the directory and a single output from the list; [‘bulbasaur’,’charmander’,’pikachu’,’squirtle’, ‘no _match’] [Task] Extrapolation of Time Based on the results of the previous task that should accessible in Python, calculated how long in seconds it would take to analyze the following size databases. Take a baseline comparison of each of the 4 OpenCV histogram matching techniques for bin_sizes : 32, 64, 128, 256, and estimate if the system needed to scan 1 Million images how long would it take? Compare the performance of each type of histogram comparison. Plot the results using Bar Chart. Show the average execution time of each technique. To do this, you may take the target image and compare to another target image and run the comparison 10 times (to average the performance).
[TASK] Graduate students – Use the subplot feature pcolor to plot multiple histogram comparisons between each set of images. type of histogram heatmaps comparison for each 10 image comparisons. For example the graph below compares 25*25 random scores that are represent having 25 items that are compared with each of the other 25. In this way, the diagonal would represent similarity scoring of identical values. This graph type can thus compare the 10 pokemon of each class. Goal: how similar (good/variant) is each set of 10 sample files in defining the class. Requirement: Make plots that visually characterize the similarity of each class. Decide which histogram style to use for comparison or come up with an objective way to characterize the similarity. An example outcome would be that if 10 images were provided for the set, but they were an identical in content, how would your system rate that as a poor training set. Requirement: in your Jupyter notebook, you must describe your findings based on the visualization or any scoring.
[All Student Rubric] Upload a single Jupyter notebook to Canvas. Your grade will be based on how well you accomplish the task of writing modular code to solve a computer vision problem. If you hack a solution that you believe doesn’t do anything wrong, but no one can use because only you can see the flow of the solution, it is not a good solution. At the end of your Jupyter notebook, the instructor should be able to pass a single Mystery image (one that is not in the mystery folder) into your framework, and return a prediction: [‘bulbasaur’,’charmander’,’pikachu’,’squirtle’, ‘no_match’] The more you have created a clean framework that is easy to understand and that all the cells are meaningful to describe your approach, the higher the grade for accomplishing the task.