What Is Relative Entropy?

1 mainIn this blog post, we will be using a bit of background from my previous blog post. If you are familiar with the basics of entropy coding, you should be fine. If not, you may want to quickly read through my previous blog post. So coming to the topic at hand, let’s continue our discussion on entropy coding. Let’s say we have a stream of English alphabets coming in, and you want to store them in the best possible way by consuming the least amount of space. So you go ahead and build your nifty entropy coder to take care of all this. But what if you don’t have access to all the data? How do you know what alphabet appears most frequently if you can’t access the full data? The problem now is that you cannot know for sure if you have chosen the best possible representation. Since you cannot wait forever, you just wait for the first ‘n’ alphabets and build your entropy coder hoping that the rest of the data will adhere to this distribution. Do we end up suffering in terms of compression by doing this? How do we measure the loss in quality?   Continue reading

What Is Entropy Coding?

1 mainEntropy Coding appears everywhere in modern digital systems. It is a fundamental building block of data compression, and data compression is pretty much needed everywhere, especially for internet, video, audio, communication, etc. Let’s consider the following scenario. You have a stream of English alphabets coming in and you want to store them in the best possible way by consuming the least amount of space. For the sake of discussion, let’s assume that they are all uppercase letters. Bear in mind that you have an empty machine which doesn’t know anything, and it understands only binary symbols i.e. 0 and 1. It will do exactly what you tell it to do, and it will need data in binary format. So what do we do here? One way would be to use numbers to represent these alphabets, right? Since there are 26 alphabets in English, we can convert them to numbers ranging from 0 to 25, and then convert those numbers into binary form. The biggest number, 25, needs 5 bits to be represented in binary form. So considering the worst case scenario, we can say that we need 5 bits to represent every alphabet. If have to store 100 alphabets, we will need 500 bits. But is that the best we can do? Are we perhaps not exploring our data to the fullest possible extent?   Continue reading

Reading JPEG Into A Byte Array

Let’s say you are workBits_and_bytesing with images for your project. A lot of times, when you have to work across multiple platforms, the encoding doesn’t remain the same. In these scenarios, you cannot process images directly by treating them like 2D matrices. One of the most common image formats you will comes across is JPEG. If you are working on the same platform using a library like OpenCV, you can directly read JPEG files into 2D data structures. If not, you will have to read it into a byte array, process it and then encode it back. How do we do that?   Continue reading

Digital Watermarking

mainLet’s say you want to verify the authenticity of a signal. The signal can take any form like an image, audio, video, or any other kind of bit stream. By now, everybody would have heard the term “watermark” being used in the general sense. The most common example would be currency notes. Watermarks are embedded to verify the authenticity of the notes. But how do we achieve that with more complicated signals? As things move into the virtual world, where the threats are elevated to a much higher and abstract level, we need a way to verify the authenticity of different forms of digital signals. How do we do it?   Continue reading

Image Steganography

mainAs discussed in my previous post, steganography is the art of hiding the fact that communication is taking place. We achieve this by hiding original information inside other information known as carrier files. Many different carrier file formats can be used, but digital images are the most popular because of their frequency of occurrence on the internet. For hiding secret information in images, there exists a large variety of steganographic techniques, some are more complex than others, and all of them have respective strong and weak points. Different applications have different requirements of the steganography technique used. For example, some applications may require absolute invisibility of the secret information, while others require a larger secret message to be hidden. How do we achieve this? How robust is it?   Continue reading

The Ramifications Of H.265

H265The International Telecommunication Union (ITU) recently approved the next generation video format known as H.265, which would be the successor to the current H.264 standard. H.265, also informally known as High Efficiency Video Coding (HEVC), is supposed to be twice as efficient as H.264 in terms of compression. H.265 relies on the fact that processing power is increasing on our devices, and uses more processing power to achieve better compression. Now how will this affect our lives? Is this just an algorithmic improvement or will it have a tangible impact?   Continue reading

Principal Component Analysis

Principal Component Analysis (PCA) is one of most useful tools in the field of pattern recognition. Let’s say you are making a list of people and collecting information about their physical attributes. Some of the more common attributes include height, weight, chest, waist and biceps. If you store 5 attributes per person, it is equivalent to storing a 5-dimensional feature vector. If you generalize it for ‘n’ different attributes, you are constructing an n-dimensional feature vector. Now you may want to analyze this data and cluster people into different categories based on these attributes. PCA comes into picture when have a set of datapoints which are multidimensional feature vectors and the dimensionality is high. If you want to analyze the patterns in our earlier example, it’s quite simple because it’s just a 5-dimensional feature vector. In real-life systems, the dimensionality is really high (often in hundreds or thousands) and it becomes very complex and time-consuming to analyze such data. What should we do now?   Continue reading