Archives

Big data is here. Go and have a cup of tea. 1

The trends are clear.  The last couple of years have seen an unprecedented interest in big data analysis.

The search for "big data" in Google Trends shows an explosion of interest since 2010.

The reasons for this are simple.  With ever expanding capacities and (Thai floods not withstanding) plummeting costs for hard drives, the amount of data that can be cheaply stored has exploded.  Combined with cheap automated collection methods, many fields have seen orders of magnitude increases in data volume that make a mockery of any hyperbole you can throw at them.

This increase in data volumes can improve the power of your statistical tests, and even allow new types of analyses that weren’t previously possible.  It’s a wonderful thing for statisticians and data scientists. The downside is that to deal with big datasets you often need to learn a whole new set of skills.  That time spent worrying about hardware, or trying to be a network administrator is time you could have spent analysing data.  At Live Analytics we believe that you should be allowed to get on with data analysis, without worrying about these other distractions.  As well as avoiding the hassle of software and package installation, Live-R has a few tricks up its sleeves to help you with big analyses.

Firstly, the architecture that Live-R is built upon is highly scalable.  If you need more memory, Live-R will allocate more memory to you.  You no longer need to go to the shops and buy another stick of RAM.

However, by far the most useful feature of Live-R to assist is big analyses is the ability to run tasks in the background, letting you get on with other work while your job executes, or even better, going to have a cup of tea.  Here’s how it works.

First you write a script to implement your analysis.  I suggest testing it on a small dataset to make sure it works correctly, before you run the full thing.  In this case, we are just generating a large matrix and calculating the QR factorisation of it.  Note that the script needs to include a  way of saving the outputs: either printing to screen, or writing to file (in this case we call save to save the variable). Now click File -> “Execute script in background”.

The first step is to create a script and click "Execute script in background" from the File menu.

Secondly, choose your script file, either from your cloud-hosted files within Live-R, or from your own machine.

The second step is to select a script file to run.

Live-R gives you a message to say that it is running, and where to find the output.  Click OK and have that cup of tea.

Live-R provides a message stating which files will be created: you get console and error output.

When it’s done, two files showing the console output and error output are created, along with the variables we saved.  It’s as easy as that.

When the analysis is complete, the generated files appear in your cloud file system.

Posted on: 06-6-2012
Posted in: Blog, Live Analytics, Live-R, R

What is Live-R? 1

Live-R screenshot

R in a browser!

Put simply, Live-R is R in a browser.  That means you get all the power of R, accessible from any internet connected device, and without having to worry about installing software on your machine. But Live-R is more than that.  At Live Analytics, we know that the best research comes out of collaboration, so Live-R lets you share your work easily.  With just a couple of clicks, you can share your R workspace, or your code with colleagues.  In fact, you can even let your colleagues edit a file at the same time as you.

In future posts, we’ll take a more detailed look at the features of Live-R.  In the meantime, here are some reasons that you might like to try it.

  • Live-R’s fully scalable web architecture means that you can analyse big data without worrying about hardware requirements.
  • No software installation means no more worries with IT support.  Those hours spent trawling the internet because a package wouldn’t install properly, or on the phone with a technician because the corporate network won’t play nicely with R will be a distant memory.
  • Sharing is easy and fun.  Got an idea for that someone might like?  Show them your code.  Stuck on a problem?  Make it easy for people to help you by showing them your problem.  Live-R makes it simple to share workspaces and code, and to publish your results.
  • You can even share with yourself!  If you have an analysis that takes a long time, you can start it on one machine, and pick up your results on another.  (There’s a batch mode feature that leaves you free to work on other problems in the meantime.)  Now you are free from being tied to a single machine.
  • Live-R also makes it easy to teach and to learn statistics, with builtin integration of course materials and textbooks.

Live-R is currently in beta, but it is ready for real-world use.  If you are interested in trying it, click here.

Posted on: 05-23-2012
Posted in: Blog, Live Analytics, Live-R

Welcome to Live Analytics Blog, Home of Live-R 0

Welcome to the Live Analytics (Live-R) blog. You can read more about the scope of Live Analytics, the industry’s first enterprise-class, cloud-based, 100% web-delivered data analytics and visualization platform.

Simply put, our goal is to remove the obstacles data analysts face with respect to high cost of ownership, ease of use and live collaborations. And do it in an open way so there is no lock-in to vendor specific implementation, platform and services.

Please sign up for Live Analytics (Live-R) here. By signing up, you will get an invitation to use the service (it is first come, first served as we scale the service) and notification when you can testdrive Live-R.

Please follow this blog’s feed to receive updates on Live-R / Live Analytics going forward. And you can follow us on Twitter @LiveAnalytics

Happy crunching,

The Live Analytics Team

Posted on: 05-19-2011
Posted in: Blog, Live Analytics, Live-R, R