How To Launch A Spark Cluster On Amazon EC2

1 mainApache Spark is marketed as “lightning fast cluster computing” and it stands true to its word! It can do amazing things really quickly using a cluster of machines. So how do we assemble that cluster? Let’s say you are using a cloud service provider like Amazon Web Services. Do we need to spin up a bunch of instances ourselves to form a “cluster”? Well, not really! Spark can launch a cluster by itself and you can control everything from one machine. You just need to log into your main instance and Spark will automatically launch all the instances in the cluster for you. It’s beautiful! Let’s go ahead and see how to launch a cluster, shall we?   Continue reading

Getting Started With Apache Spark In Python

1 mainIn one of the previous blog posts, we discussed how to get Apache Spark up and running on your Ubuntu box. In this post, we will start exploring it. One of the best things about Spark is that it comes with a Python API that works like a charm! The API also available in Java, Scala, and R. That pretty much covers the entire world of programming and data science! Spark’s shell provides a great way to analyze our data and work with it interactively. We are going to see how to interact with Spark Python API in this post. You would have downloaded Spark on your machine. Let’s go into “spark-1.5.1” directory on your terminal and get started, shall we?   Continue reading

How To Install Apache Spark On Ubuntu

1 mainThere’s so much data being generated in today’s world that we need platforms and frameworks that it’s mind boggling. This field of study is called Big Data Analysis. With so much data lying around, often ranging in petabytes and exabytes, we need super powerful systems to process it. Not only that, we need to do it high efficiency. If you try to do it using your regular ways, you will never be able to do anything in time, let alone doing it in real-time. This is where Apache Spark comes into picture. It is an open source big data processing framework that can process massive amounts of data at high speed using cluster computing. Let’s see how we can install it on Ubuntu.   Continue reading

Why Would We Ever Use Blind Search?

mainOver the last few decades, we have seen a lot of technologies come by and make a significant impact. Most of these technologies, if you have noticed, revolve around intelligent actions. Let’s say you are in the middle of a street and you want a cab. We can solve this problem in a couple of different ways depending on the level of intelligence we put into our solution. How can we design something that can make use of all the data and provide the best possible solution to the person in the middle of the street? We can say that intelligent action involves search in some way. Searching is needed in a variety of situations, so developing mathematically and computationally strong algorithms is an absolute must. In most of the real life situations, we don’t really know where to look or how a particular search is going to pan out. How do we formulate this?   Continue reading

Quantum Encryption And Black Holes – Part 2/2

1 mainIn the previous post, we discussed about the concepts of quantum encryption and black holes. We also talked about how we do cryptography in the subatomic world. This blog post is a continuation of that discussion. As the title suggests, the overarching theme is the relationship between quantum encryption and black holes. Let’s continue talking about it then. Although quantum encryption looks extremely robust in theory, how practical is it? What do we know about its security and how is it related to black holes? We know that nothing can escape from black holes, so we need a way to understand more about the black holes.   Continue reading

Quantum Encryption And Black Holes – Part 1/2

1 mainIs that really the title? It looks like two random things mashed up together. Doesn’t make much sense, right? Well, recent research suggests that quantum encryption and black holes may be related. A proposed mathematical proof outlines the way in which information behaves in coded messages, and this may have implications for black holes. The proof basically suggests that the radiation spit out by black holes may retain information about them. The research not only focuses on encoding communications in quantum mechanical systems, but also addresses a long-standing question for theoretical physicists: What exactly happens to all the stuff that falls into a black hole? Is it possible to retrieve any information about the black hole?   Continue reading

Quantum Computing And Machine Learning

mainQuantum Computing refers to the use of quantum mechanical phenomena to make computations. This field is making big strides in the last decade because it can actually help us solve some of the most challenging problems in the realm of computer science, particularly in machine learning and security. Machine learning is all about building better models of the world to make more accurate predictions and security is about safeguarding the things we have built. For example, if we want the machines to see things better, we need better models of how we process visual data. If we want to understand currency fluctuations, we need better models of how they change over time. If we want to create effective environmental policies, we need better models of what’s happening to our climate. So how can we use quantum computing to do these things?   Continue reading