We deal with real time data all the time. If you look at those analytics dashboards, you can see how they perform computations and tell us what happened in the last 60 mins or may be the last 7 hours. They are dealing with terabytes of data and yet they can process all of that … Continue reading Performing Windowed Computations On Streaming Data Using Spark In Python
Category: Programming
Analyzing Real-time Data With Spark Streaming In Python
There is a lot of data being generated in today's digital world, so there is a high demand for real time data analytics. This data usually comes in bits and pieces from many different sources. It can come in various forms like words, images, numbers, and so on. Twitter is a good example of words being … Continue reading Analyzing Real-time Data With Spark Streaming In Python
Launching A Spark Standalone Cluster
In the previous blog post, we saw how to start a Spark cluster on EC2 using the inbuilt launch scripts. This is good if you want get something up and running quickly, but it won't allow fine-grained control over our cluster. A lot of times, we would want to customize the machines that we spin up. Let's say … Continue reading Launching A Spark Standalone Cluster
How To Launch A Spark Cluster On Amazon EC2
Apache Spark is marketed as "lightning fast cluster computing" and it stands true to its word! It can do amazing things really quickly using a cluster of machines. So how do we assemble that cluster? Let's say you are using a cloud service provider like Amazon Web Services. Do we need to spin up a … Continue reading How To Launch A Spark Cluster On Amazon EC2
Getting Started With Apache Spark In Python
In one of the previous blog posts, we discussed how to get Apache Spark up and running on your Ubuntu box. In this post, we will start exploring it. One of the best things about Spark is that it comes with a Python API that works like a charm! The API also available in Java, Scala, and … Continue reading Getting Started With Apache Spark In Python
Understanding Filter, Map, And Reduce In Python
Even though lot of people use Python in an object oriented style, it has several functions that enable functional programming. For those of you who don't know, functional programming is a programming paradigm based on lambda calculus that treats computation as an evaluation of mathematical functions. Some of prominent functional programming languages include Scala, Haskell, … Continue reading Understanding Filter, Map, And Reduce In Python
How To Install Apache Spark On Ubuntu
There's so much data being generated in today's world that we need platforms and frameworks that it's mind boggling. This field of study is called Big Data Analysis. With so much data lying around, often ranging in petabytes and exabytes, we need super powerful systems to process it. Not only that, we need to do … Continue reading How To Install Apache Spark On Ubuntu
Enabling Tab Autocomplete In Python Shell
It's fun to play around with Python. One of its best features is the interactive shell where we can experiment all we want. Let's say you open up a shell, declare a bunch of variables and want to operate on them. You don't want to type the full variables names over and over again, right? Also, it's difficult … Continue reading Enabling Tab Autocomplete In Python Shell
How To Add Swap Space On Ubuntu
Whenever you are building an application that's memory intensive, you are bound to run into memory issues. Those out of memory errors are painful to deal with, especially when they happen during production. Before putting your code on your server, you need to make sure that it can handle the application's memory requirements. But even … Continue reading How To Add Swap Space On Ubuntu
What Is External Sorting?
Sorting is one of the most common things we do in programming. We are given a bunch of numbers and we want to arrange them according to some rule. Let's say we want to arrange them in ascending order. To sort these numbers, people tend to use a sorting algorithm that takes place entirely within … Continue reading What Is External Sorting?
Perpetual Enigma 