Posts Tagged ‘Animation’

Further analysis of the scraped BBC data

May 4, 2011

Further to my previous post, I have been mining the data I collected to see if I could find anything of interest, and evaluate the data I have collected.

One of things that interested to me, was to look at the placement of the story compared to it’s position in the top ten over time. So I set off my program to scrape the data from 9AM through to 9PM, I didn’t want too much data to start with. Scraping this data every hour. I then spent a while writing SQL statements to capture the statistics I was looking for over the time period, below you can see a simple line graph (it is in Logarithmic scale):

I produced this using Open Office.

I find it is quite interesting to see the third position on the indexes doesn’t get utilised so much. You could also suggest as the day gets later the more people start reading other stories, i.e. features, second, and other stories. However, one can’t jump to any conclusion for two reasons which jump out at me:

1 – This sample set is way too small!
2 – I think I need more detail, it would be interesting (for example) which position of the features and analysis are being selected? The same goes for the other stories.

I decided I would cobble together a simple heat map of the index, aggregated, over the time period to see the “behaviour”. Feel free to take a look, personally I feel it re-enforces my previous points (I used JavaFx script 1.3 via Netbeans to produce this):

I think I might go back to the scraper program and see if I can expand the data gathered.

Trying to Replicate Processing’s House of Cards code in JavaFX

August 12, 2010

One of the books I am reading, on and off, is Beautiful data. I recently read Chapter 10 – Building Radiohead’s House of Cards by Aaron Koblin with Valdin Klump and thoroughly enjoyed it. I was mightily impressed that Processing was used in the production of the video, which gave me the inspriation to:

  1. Have a look at the Processing code
  2. See if I can replicate the code in JavaFX!

In the first instance you can find the Processing code, which is part of the Google project, here. I encourage you to check it out, it is really good (especially after reading the chapter in the book!).

Ok, so the code for Processing is really straight forward (I have stripped out comments):

import processing.opengl.*;

int frameCounter =1;

void setup()
{
  size(1024,768, OPENGL);

  strokeWeight(1);
  
}
void draw()
{
  background(0);
  translate(width/2, height/2); 
  translate(-150,-150);
  scale(2);
  
  String[] raw = loadStrings(frameCounter+".csv");

  for(int i = 0; i 2101)
  {
    exit();
    println("done");  
  }
}

With the data in place (1000 CSV files), this works really nicely “straight out of the box”. It is a joy to play with it.

Here is a screen shot of the data represented in Processing:

So, being thoroughly motivated I started work on writing the JavaFX equivalent. In the first instance I needed to load the CSV files. Now Processing has a function for that, for JavaFx I decided to write my own Java class to process it:

public class LoadStrings
{
    URL url;
    FileInputStream fr;
    BufferedReader br;
    List<String> myLines = new ArrayList<String>();
    String a;
    
    public LoadStrings(String fname)
    {
        String theLine;
        try
        {
            url = new URL(fname);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            while((theLine = br.readLine()) != null)
            {
                myLines.add(theLine);
            }

            br.close();

        }
        catch(IOException e)
        {
            System.out.println(e.toString());
        }
        catch(Exception e)
        {
            System.out.println("It ain't worked!");
            System.out.println(e.toString());
        }
    }

    public String[] getRows()
    {
        String[] res = new String[myLines.size()];
        for (int i = 0; i < myLines.size(); i++)
        {
            res[i] = myLines.get(i);
        }

        return res;
    }
}

With that complete I started on the Main application. Processing’s draw() function is a continuous loop, I believe based on the framerate. With this in mind I needed to replicate the draw function, so I choose to use an animator – Timeline. Before defining this animation I declared:

var lines : Line[] = [];
var theFrame = 0;

I defined my Timeline as:

def projector : Timeline = Timeline
{
    repeatCount: 1000
    framerate: 60

    keyFrames:
    [
        KeyFrame
        {
            time: 0.5s
            action: function()
            {
                delete lines;
                var dataPoints : DataPoint[] = [];

                var f: String = "{__DIR__}data/{theFrame+1}.csv";

                var fr = new LoadStrings(f);

                var rows : String[] = fr.getRows();

                var counter = 0;

                while (counter < rows.size())
                {
                    var thisLine : String[] = rows.get(counter).split(',');

                    var x = Number.parseFloat(thisLine[0]) + (1024/2) + -200;
                    var y = Number.parseFloat(thisLine[1]) + (768/2) + -200;
                    var sr = (Number.parseFloat(thisLine[3]) / (255.0 * 1.1) * (255.0));
                    var sg = (Number.parseFloat(thisLine[3]) / (255.0 * 1.6) * (255.0));

                    insert DataPoint
                    {
                        x: x
                        y: y
                        z: Number.parseFloat(thisLine[2]);
                        intensity: Number.parseFloat(thisLine[3]);
                        scaledRed: sr
                        scaledGreen: sg
                    } into dataPoints;
                    counter++;
                }

                for (dp in dataPoints)
                {
                    insert Line
                    {
                        startX: dp.x
                        endX: dp.x+1
                        startY: dp.y
                        endY: dp.y+1
                        stroke: Color.rgb(dp.scaledRed, dp.scaledGreen, 200,1);
                    } into lines
                }
                theFrame++;
            }
        }
    ]
}

projector.play();

So my stage looks like this:

var theStage : Stage = Stage
{
    title: "House of Cards"
    scene: Scene
    {
        width: 1024
        height: 768
        fill: Color.BLACK;
        content: bind
        [
            lines
        ]
    }
}

It is all very exciting, isn’t it…? Before implementing this code I managed to load the application showing just the first file and it’s data points and it looked exceptional:

I was very excited, although a little apprehensive as the time it took to both compile (over 2 minutes on my machine with the full 1000 files) and running the app. It was a lot slower to load than the Processing instance. I am writing this in Netbeans, on OS X, with 4GB of RAM.

Running this, with time: 0.5s results in a slow animation, which leaves you wanting more. I tried to reduce the time, even commenting it out, but the image couldn’t get updated in time and the effect was even worse.

Conclusion:

Ok, so JavaFX was not really designed for this type of work (well that is my understanding). However I am a little surprised that I can’t run this like I can with Processing. I even tried increasing the memory setting in Netbeans by changing the properties of the project so that Run JVM arguments are: -Xmx3072m.

This is my first attempt at anything like this, so perhaps I am mis-understanding something somewhere.

Perhaps I am misunderstanding animations in JavaFX?

I tried to look into the 3D functionality with JavaFx, but I struggled to find examples I could relate to.

All in all I feel I have tried (and there is never any harm in trying). I like what I have, I have feeling it could be better?

Any pointers / thoughts?

Update: 15th August

So after some excellent feedback from Jonathan Giles, I have changed the code as follows.

The LoadStrings now looks like:

public class LoadStrings
{
    URL url;
    FileInputStream fr;
    BufferedReader br;
    List<String> myLines = new ArrayList<String>();
    List<DataPointJ> dataPoints = new ArrayList<DataPointJ>();   

    public LoadStrings(String fname)
    {
        String theLine;
        try
        {
            url = new URL(fname);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            while((theLine = br.readLine()) != null)
            {
                myLines.add(theLine);
                String[] t = theLine.split(",");
                dataPoints.add(new DataPointJ(Float.parseFloat(t[0]),Float.parseFloat(t[1]),Float.parseFloat(t[2]),Integer.parseInt(t[3])));
            }

            br.close();

        }
        catch(IOException e)
        {
            System.out.println(e.toString());
        }
        catch(Exception e)
        {
            System.out.println("It ain't worked!");
            System.out.println(e.toString());
        }
    }

    public String[] getRows()
    {
        String[] res = new String[myLines.size()];
        for (int i = 0; i < myLines.size(); i++)
        {
            res[i] = myLines.get(i);
        }

        return res;
    }

    public List<DataPointJ> getDps()
    {
        return dataPoints;
    }
}

The Timeline:

var projector : Timeline = Timeline
{
    repeatCount: 50

    keyFrames:
    [
        KeyFrame
        {
            time: 0.5s
            action: function()
            {
                var f: String = "{__DIR__}data/{theFrame+1}.csv";

                var fr = new LoadStrings(f);

                var dps = fr.getDps();

                lines = for ( i in dps)
                {
		    		Line
                    {
						startX: i.getX();
						endX: i.getX()+1;
                        startY: i.getY();
                        endY: i.getY()+1;
                        stroke: Color.rgb(i.getScaledRed(), i.getScaledGreen(), 200, 1);
                    }
                }
                theFrame++;
            }
        }
    ]
}

projector.play();

This saw improvements, however I could not run the timeline with less than 0.5s. I decided, with the new changes I could load the data into memory. I realsied I would have to do this incrementally, so I decided to define 10 sequences which would each hold 50 instance of the LoadStrings. Then I changed the animation which would resolve each LoadStrings, get the data and build the line. For each 50 it would move onto the next sequence and delete the previous sequence:

function loadDataIntoMemory():Void
{
    for (tf in [0..499])
    {
        var f: String = "{__DIR__}data/{tf+1}.csv";

        var fr = new LoadStrings(f);
        if (tf <50) insert fr into filePointer1;
        if (tf >50 and tf < 101) insert fr into filePointer2;
        if (tf > 100 and tf < 151) insert fr into filePointer3;
        if (tf > 150 and tf < 201) insert fr into filePointer4;
        if (tf > 200 and tf < 251) insert fr into filePointer5;
        if (tf > 250 and tf < 301) insert fr into filePointer6;
        if (tf > 300 and tf < 351) insert fr into filePointer7;
        if (tf > 350 and tf < 401) insert fr into filePointer8;
        if (tf > 400 and tf < 451) insert fr into filePointer9;
        if (tf > 450 and tf < 501) insert fr into filePointer10;

    }
}

The Timeline:

var projector4 : Timeline = Timeline
{
    repeatCount: 500
    framerate: 60

    keyFrames:
    [
        KeyFrame
        {
            action: function()
            {
                var fr;
                if (theFrame < 50) fr = filePointer1[theFrame];
                if (theFrame >= 50 and theFrame < 100)
                {
                    if (sizeof filePointer1 > 0)
                    {
                        delete filePointer1;
                    }
                    fr = filePointer2[theFrame-50];

                }
                if (theFrame >= 100 and theFrame < 150)
                {
                    if (sizeof filePointer2 > 0) delete filePointer2;

                    fr = filePointer3[theFrame-100];
                }
                if (theFrame >= 150 and theFrame < 200)
                {
                    if (sizeof filePointer3 > 0) delete filePointer3;

                    fr = filePointer4[theFrame-150];
                }
                if (theFrame >= 200 and theFrame < 250)
                {
                    if (sizeof filePointer4 > 0) delete filePointer4;

                    fr = filePointer5[theFrame-200];
                }
                if (theFrame >= 250 and theFrame < 300)
                {
                    if (sizeof filePointer5 > 0) delete filePointer5;

                    fr = filePointer6[theFrame-250];
                }
                if (theFrame >= 300 and theFrame < 350)
                {
                    if (sizeof filePointer6 > 0) delete filePointer6;

                    fr = filePointer7[theFrame-300];
                }
                if (theFrame >= 350 and theFrame < 400)
                {
                    if (sizeof filePointer7 > 0) delete filePointer7;

                    fr = filePointer8[theFrame-350];
                }
                if (theFrame >= 400 and theFrame < 450)
                {
                    if (sizeof filePointer8 > 0) delete filePointer8;

                    fr = filePointer9[theFrame-400];
                }
                if (theFrame >= 450 and theFrame < 500)
                {
                    if (sizeof filePointer9 > 0) delete filePointer9;

                    fr = filePointer10[theFrame-450];
                }
                var dps = fr.getDps();

                lines = for ( i in dps)
                {
                    Line
                    {
                        startX: i.getX();
                        endX: i.getX()+1;
                        startY: i.getY();
                        endY: i.getY()+1;
                        stroke: Color.rgb(i.getScaledRed(), i.getScaledGreen(), 200, 1);
                    }
                }

                theFrame++;
            }

        }

    ]
}

This allowed the script to run, for the first time, through 500 files showing the animation. It gobbles up the memory, and I know that this code can be improved, however I am not going to have any time this week so I thought I would post this update for the moment.

The UK Economy, JavaFX and a Dashboard…

April 5, 2010

I was recently looking for a new idea to exercise my JavaFX skills, and with the economic wows which have affected so many, I wanted to look at how I could represent the last decade’s economic data WITHOUT box charts, histograms, and line graphs! I decided that a dashboard of the data might prove interesting to create, and could be visually “exciting” (I use the word tentatively with this type of data). I wanted to try and represent this data around, say, an old car dashboard which use to be full of dials and knobs. Unfortunately you can’t tap these dials to get a “better” reading like I use to do all those years ago.

First of, where did I source the data? Well it is all freely available and the first place to look is data.gov.uk. This is an extremely useful resource for all you data junkies and I have been a member from earlier this year (before it was live). It is amazing what data is available here, however I do find it a little cumbersome being directed from one website to another to try and get the data I want and then having to re-issue my query. Thankfully the Guardian have a data blog, where you can source a lot of the data available at data.gov.uk but without having to pull your hair out! I was able to source data from both, thanks 🙂

The data I am representing is: Inflation, Interest and unemployment rates (these are monthly), and then the deficit and GDP for the UK (these are yearly). I decided I would represent the first three with dials which would move to the relevant value as each month passed, and the latter two I would represent with gauges which would move up and down depending on the values.

After some searching for inspiration I found a perfect tutorial from which I could grow my idea. So with the three dials defined and all reading their data independently (each dial is its own task, see my previous post for more info). The two gauges, similar to temperature and fuel, with a pointer moving up and down didn’t have to be set up as tasks, I could define them as triggers which would be actioned each time a year clocked over.


UK Economic Dashboard

What do think?

I hope you find the data representation and animation an original idea? I have never seen it represented like this before? I am off to source some more recent data…. I won’t pass opinion on the data itself, it is too close to the election to get drawn into that!

JavaFX and Collatz Conjecture…

March 23, 2010

Following on from my previous posting about multi-threading I wanted to share with you a little “toy” I recently wrote. Upon understanding how to implement threading into JavaFX I wanted to write up an example to help me get a good understanding. I decided I would visualise Collatz Conjecture, feeling it was a perfectly simple principle that I could animate.

Firstly, for those of you who are not aware, Collatz Conjecture is nice and simple to perform. You take any natural number (N) and if it is odd then multiple it 3 and add 1, and if it is even divide by 2. This continues until you reach the number 1, below is a table of two examples for the natural numbers 4 and 5:

N = 4 N = 5
4 is even so 4/2. N = 2 5 is odd so 5*3 + 1. N = 16
2 is even so 2/2. N = 1 16 is even so 16/2. N = 8
8 is even so 8/2. N = 4
4 is even so 4/2. N = 2
2 is even so 2/1. N = 1

It isn’t really rocket science is it? It is quite interesting though. Try the number 27. There is more information on this available at, unsurprisingly, Wikipedia.

With that dealt with lets go back to the animation. I decided I wanted to animate this process, in fact I wanted each number to roll across the screen as it iterated through all of the numbers. As this idea progressed in my mind I happened across the thought that I could make it like a pack of cards being dealt. However the twist was the area of the screen number would land in would be determined by its value:

The final positioning (X,Y) of each rectangle, or card, is down to random positioning. This would give you a rather crude distribution.

Each card will rotate across the screen to its destination, one after the other (as if it had been dealt their). Hence I employed the threading logic as defined in my previous post, with each toss of the rectangle taking around 0.5 seconds to rotate and position itself across the screen.

All you, the user, need do is enter a number, watch the numbers being dealt across the screen and titter at the simplisity, oh and select again if you want to watch another number 🙂

Have a look, what do you think?

If anyone is interested I could go into more detail about the code?

JavaFX Applet