The Internet of Things (IoT) ecosystem is rapidly expanding. Some analysts predict that there will be around 50 billion connected devices by 2020. If you are new to IoT, it refers to the collective ecosystem of devices that are connected to the internet. These devices can be sensors, actuators, health monitors, meters, and so on. What did people do before IoT? Well, they had devices that weren’t connected to the internet. Hence it was difficult to monitor and analyze data in real time. This meant that people were leaving a lot of interesting data unused, which directly translates to lost revenue of billions of dollars. By connecting all the devices to the internet, we are enabling ourselves to take actions in real time. It’s obvious that device connectivity is a really important aspect in IoT. How do we ensure connectivity? How can we enable low cost hardware devices to communicate with the cloud without expensive processors? Continue reading “Undestanding IoT Gateways”
Tag: Big Data
Performing Windowed Computations On Streaming Data Using Spark In Python
We deal with real time data all the time. If you look at those analytics dashboards, you can see how they perform computations and tell us what happened in the last 60 mins or may be the last 7 hours. They are dealing with terabytes of data and yet they can process all of that in real time. These insights are extremely valuable because you can take the right actions if you know what’s happening. If you have a shopping website, you need to know what happened in the last few hours so that you can boost your sales. Are there a lot of visitors from France? Can I organize a quick French themed promotion to increase my sales during peak hours? The answers to all these lies deep within your data. Spark Streaming is amazing at these things! So how do we do windowed computations in Spark? How can we process this data in real time? Continue reading “Performing Windowed Computations On Streaming Data Using Spark In Python”
Analyzing Real-time Data With Spark Streaming In Python
There is a lot of data being generated in today’s digital world, so there is a high demand for real time data analytics. This data usually comes in bits and pieces from many different sources. It can come in various forms like words, images, numbers, and so on. Twitter is a good example of words being generated in real time. We also have websites where statistics like number of visitors, page views, and so on are being generated in real time. There are so much data that it is not very useful in its raw form. We need to process it and extract insights from it so that it becomes useful. This is where Spark Streaming comes into the picture! It is exceptionally good at processing real time data and it is highly scalable. It can process enormous amounts of data in real time without skipping a beat. So how exactly does Spark do it? How do we use it? Continue reading “Analyzing Real-time Data With Spark Streaming In Python”
Getting Started With Apache Spark In Python
In one of the previous blog posts, we discussed how to get Apache Spark up and running on your Ubuntu box. In this post, we will start exploring it. One of the best things about Spark is that it comes with a Python API that works like a charm! The API also available in Java, Scala, and R. That pretty much covers the entire world of programming and data science! Spark’s shell provides a great way to analyze our data and work with it interactively. We are going to see how to interact with Spark Python API in this post. You would have downloaded Spark on your machine. Let’s go into “spark-1.5.1” directory on your terminal and get started, shall we? Continue reading “Getting Started With Apache Spark In Python”
How To Install Apache Spark On Ubuntu
There’s so much data being generated in today’s world that we need platforms and frameworks that it’s mind boggling. This field of study is called Big Data Analysis. With so much data lying around, often ranging in petabytes and exabytes, we need super powerful systems to process it. Not only that, we need to do it high efficiency. If you try to do it using your regular ways, you will never be able to do anything in time, let alone doing it in real-time. This is where Apache Spark comes into picture. It is an open source big data processing framework that can process massive amounts of data at high speed using cluster computing. Let’s see how we can install it on Ubuntu. Continue reading “How To Install Apache Spark On Ubuntu”
Perpetual Enigma 