Posts Tagged ‘JavaFX’

Further analysis of the scraped BBC data

May 4, 2011

Further to my previous post, I have been mining the data I collected to see if I could find anything of interest, and evaluate the data I have collected.

One of things that interested to me, was to look at the placement of the story compared to it’s position in the top ten over time. So I set off my program to scrape the data from 9AM through to 9PM, I didn’t want too much data to start with. Scraping this data every hour. I then spent a while writing SQL statements to capture the statistics I was looking for over the time period, below you can see a simple line graph (it is in Logarithmic scale):

I produced this using Open Office.

I find it is quite interesting to see the third position on the indexes doesn’t get utilised so much. You could also suggest as the day gets later the more people start reading other stories, i.e. features, second, and other stories. However, one can’t jump to any conclusion for two reasons which jump out at me:

1 – This sample set is way too small!
2 – I think I need more detail, it would be interesting (for example) which position of the features and analysis are being selected? The same goes for the other stories.

I decided I would cobble together a simple heat map of the index, aggregated, over the time period to see the “behaviour”. Feel free to take a look, personally I feel it re-enforces my previous points (I used JavaFx script 1.3 via Netbeans to produce this):

I think I might go back to the scraper program and see if I can expand the data gathered.

Advertisements

Trying to Replicate Processing’s House of Cards code in JavaFX

August 12, 2010

One of the books I am reading, on and off, is Beautiful data. I recently read Chapter 10 – Building Radiohead’s House of Cards by Aaron Koblin with Valdin Klump and thoroughly enjoyed it. I was mightily impressed that Processing was used in the production of the video, which gave me the inspriation to:

  1. Have a look at the Processing code
  2. See if I can replicate the code in JavaFX!

In the first instance you can find the Processing code, which is part of the Google project, here. I encourage you to check it out, it is really good (especially after reading the chapter in the book!).

Ok, so the code for Processing is really straight forward (I have stripped out comments):

import processing.opengl.*;

int frameCounter =1;

void setup()
{
  size(1024,768, OPENGL);

  strokeWeight(1);
  
}
void draw()
{
  background(0);
  translate(width/2, height/2); 
  translate(-150,-150);
  scale(2);
  
  String[] raw = loadStrings(frameCounter+".csv");

  for(int i = 0; i 2101)
  {
    exit();
    println("done");  
  }
}

With the data in place (1000 CSV files), this works really nicely “straight out of the box”. It is a joy to play with it.

Here is a screen shot of the data represented in Processing:

So, being thoroughly motivated I started work on writing the JavaFX equivalent. In the first instance I needed to load the CSV files. Now Processing has a function for that, for JavaFx I decided to write my own Java class to process it:

public class LoadStrings
{
    URL url;
    FileInputStream fr;
    BufferedReader br;
    List<String> myLines = new ArrayList<String>();
    String a;
    
    public LoadStrings(String fname)
    {
        String theLine;
        try
        {
            url = new URL(fname);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            while((theLine = br.readLine()) != null)
            {
                myLines.add(theLine);
            }

            br.close();

        }
        catch(IOException e)
        {
            System.out.println(e.toString());
        }
        catch(Exception e)
        {
            System.out.println("It ain't worked!");
            System.out.println(e.toString());
        }
    }

    public String[] getRows()
    {
        String[] res = new String[myLines.size()];
        for (int i = 0; i < myLines.size(); i++)
        {
            res[i] = myLines.get(i);
        }

        return res;
    }
}

With that complete I started on the Main application. Processing’s draw() function is a continuous loop, I believe based on the framerate. With this in mind I needed to replicate the draw function, so I choose to use an animator – Timeline. Before defining this animation I declared:

var lines : Line[] = [];
var theFrame = 0;

I defined my Timeline as:

def projector : Timeline = Timeline
{
    repeatCount: 1000
    framerate: 60

    keyFrames:
    [
        KeyFrame
        {
            time: 0.5s
            action: function()
            {
                delete lines;
                var dataPoints : DataPoint[] = [];

                var f: String = "{__DIR__}data/{theFrame+1}.csv";

                var fr = new LoadStrings(f);

                var rows : String[] = fr.getRows();

                var counter = 0;

                while (counter < rows.size())
                {
                    var thisLine : String[] = rows.get(counter).split(',');

                    var x = Number.parseFloat(thisLine[0]) + (1024/2) + -200;
                    var y = Number.parseFloat(thisLine[1]) + (768/2) + -200;
                    var sr = (Number.parseFloat(thisLine[3]) / (255.0 * 1.1) * (255.0));
                    var sg = (Number.parseFloat(thisLine[3]) / (255.0 * 1.6) * (255.0));

                    insert DataPoint
                    {
                        x: x
                        y: y
                        z: Number.parseFloat(thisLine[2]);
                        intensity: Number.parseFloat(thisLine[3]);
                        scaledRed: sr
                        scaledGreen: sg
                    } into dataPoints;
                    counter++;
                }

                for (dp in dataPoints)
                {
                    insert Line
                    {
                        startX: dp.x
                        endX: dp.x+1
                        startY: dp.y
                        endY: dp.y+1
                        stroke: Color.rgb(dp.scaledRed, dp.scaledGreen, 200,1);
                    } into lines
                }
                theFrame++;
            }
        }
    ]
}

projector.play();

So my stage looks like this:

var theStage : Stage = Stage
{
    title: "House of Cards"
    scene: Scene
    {
        width: 1024
        height: 768
        fill: Color.BLACK;
        content: bind
        [
            lines
        ]
    }
}

It is all very exciting, isn’t it…? Before implementing this code I managed to load the application showing just the first file and it’s data points and it looked exceptional:

I was very excited, although a little apprehensive as the time it took to both compile (over 2 minutes on my machine with the full 1000 files) and running the app. It was a lot slower to load than the Processing instance. I am writing this in Netbeans, on OS X, with 4GB of RAM.

Running this, with time: 0.5s results in a slow animation, which leaves you wanting more. I tried to reduce the time, even commenting it out, but the image couldn’t get updated in time and the effect was even worse.

Conclusion:

Ok, so JavaFX was not really designed for this type of work (well that is my understanding). However I am a little surprised that I can’t run this like I can with Processing. I even tried increasing the memory setting in Netbeans by changing the properties of the project so that Run JVM arguments are: -Xmx3072m.

This is my first attempt at anything like this, so perhaps I am mis-understanding something somewhere.

Perhaps I am misunderstanding animations in JavaFX?

I tried to look into the 3D functionality with JavaFx, but I struggled to find examples I could relate to.

All in all I feel I have tried (and there is never any harm in trying). I like what I have, I have feeling it could be better?

Any pointers / thoughts?

Update: 15th August

So after some excellent feedback from Jonathan Giles, I have changed the code as follows.

The LoadStrings now looks like:

public class LoadStrings
{
    URL url;
    FileInputStream fr;
    BufferedReader br;
    List<String> myLines = new ArrayList<String>();
    List<DataPointJ> dataPoints = new ArrayList<DataPointJ>();   

    public LoadStrings(String fname)
    {
        String theLine;
        try
        {
            url = new URL(fname);
            br = new BufferedReader(new InputStreamReader(url.openStream()));
            while((theLine = br.readLine()) != null)
            {
                myLines.add(theLine);
                String[] t = theLine.split(",");
                dataPoints.add(new DataPointJ(Float.parseFloat(t[0]),Float.parseFloat(t[1]),Float.parseFloat(t[2]),Integer.parseInt(t[3])));
            }

            br.close();

        }
        catch(IOException e)
        {
            System.out.println(e.toString());
        }
        catch(Exception e)
        {
            System.out.println("It ain't worked!");
            System.out.println(e.toString());
        }
    }

    public String[] getRows()
    {
        String[] res = new String[myLines.size()];
        for (int i = 0; i < myLines.size(); i++)
        {
            res[i] = myLines.get(i);
        }

        return res;
    }

    public List<DataPointJ> getDps()
    {
        return dataPoints;
    }
}

The Timeline:

var projector : Timeline = Timeline
{
    repeatCount: 50

    keyFrames:
    [
        KeyFrame
        {
            time: 0.5s
            action: function()
            {
                var f: String = "{__DIR__}data/{theFrame+1}.csv";

                var fr = new LoadStrings(f);

                var dps = fr.getDps();

                lines = for ( i in dps)
                {
		    		Line
                    {
						startX: i.getX();
						endX: i.getX()+1;
                        startY: i.getY();
                        endY: i.getY()+1;
                        stroke: Color.rgb(i.getScaledRed(), i.getScaledGreen(), 200, 1);
                    }
                }
                theFrame++;
            }
        }
    ]
}

projector.play();

This saw improvements, however I could not run the timeline with less than 0.5s. I decided, with the new changes I could load the data into memory. I realsied I would have to do this incrementally, so I decided to define 10 sequences which would each hold 50 instance of the LoadStrings. Then I changed the animation which would resolve each LoadStrings, get the data and build the line. For each 50 it would move onto the next sequence and delete the previous sequence:

function loadDataIntoMemory():Void
{
    for (tf in [0..499])
    {
        var f: String = "{__DIR__}data/{tf+1}.csv";

        var fr = new LoadStrings(f);
        if (tf <50) insert fr into filePointer1;
        if (tf >50 and tf < 101) insert fr into filePointer2;
        if (tf > 100 and tf < 151) insert fr into filePointer3;
        if (tf > 150 and tf < 201) insert fr into filePointer4;
        if (tf > 200 and tf < 251) insert fr into filePointer5;
        if (tf > 250 and tf < 301) insert fr into filePointer6;
        if (tf > 300 and tf < 351) insert fr into filePointer7;
        if (tf > 350 and tf < 401) insert fr into filePointer8;
        if (tf > 400 and tf < 451) insert fr into filePointer9;
        if (tf > 450 and tf < 501) insert fr into filePointer10;

    }
}

The Timeline:

var projector4 : Timeline = Timeline
{
    repeatCount: 500
    framerate: 60

    keyFrames:
    [
        KeyFrame
        {
            action: function()
            {
                var fr;
                if (theFrame < 50) fr = filePointer1[theFrame];
                if (theFrame >= 50 and theFrame < 100)
                {
                    if (sizeof filePointer1 > 0)
                    {
                        delete filePointer1;
                    }
                    fr = filePointer2[theFrame-50];

                }
                if (theFrame >= 100 and theFrame < 150)
                {
                    if (sizeof filePointer2 > 0) delete filePointer2;

                    fr = filePointer3[theFrame-100];
                }
                if (theFrame >= 150 and theFrame < 200)
                {
                    if (sizeof filePointer3 > 0) delete filePointer3;

                    fr = filePointer4[theFrame-150];
                }
                if (theFrame >= 200 and theFrame < 250)
                {
                    if (sizeof filePointer4 > 0) delete filePointer4;

                    fr = filePointer5[theFrame-200];
                }
                if (theFrame >= 250 and theFrame < 300)
                {
                    if (sizeof filePointer5 > 0) delete filePointer5;

                    fr = filePointer6[theFrame-250];
                }
                if (theFrame >= 300 and theFrame < 350)
                {
                    if (sizeof filePointer6 > 0) delete filePointer6;

                    fr = filePointer7[theFrame-300];
                }
                if (theFrame >= 350 and theFrame < 400)
                {
                    if (sizeof filePointer7 > 0) delete filePointer7;

                    fr = filePointer8[theFrame-350];
                }
                if (theFrame >= 400 and theFrame < 450)
                {
                    if (sizeof filePointer8 > 0) delete filePointer8;

                    fr = filePointer9[theFrame-400];
                }
                if (theFrame >= 450 and theFrame < 500)
                {
                    if (sizeof filePointer9 > 0) delete filePointer9;

                    fr = filePointer10[theFrame-450];
                }
                var dps = fr.getDps();

                lines = for ( i in dps)
                {
                    Line
                    {
                        startX: i.getX();
                        endX: i.getX()+1;
                        startY: i.getY();
                        endY: i.getY()+1;
                        stroke: Color.rgb(i.getScaledRed(), i.getScaledGreen(), 200, 1);
                    }
                }

                theFrame++;
            }

        }

    ]
}

This allowed the script to run, for the first time, through 500 files showing the animation. It gobbles up the memory, and I know that this code can be improved, however I am not going to have any time this week so I thought I would post this update for the moment.

JavaFx used to represent BP Oil disaster news timeline

July 25, 2010

Outside of following the on-going Gulf Oil Spill, I have been playing around with JavaFX and different ways of displaying on-going information. I wanted a clean way to represent this information, and decided to try and create an interactive timeline of the news stories created around this event:

This is based on the BBC’s timeline index, where I have taken each story and mapped it to the relevant position on the timeline. Further to this, I grabbed each stories text and saved it into the XML file which the JavaFx script parses. When you hoover your mouse over an event, the text (or image) of the story “pops” up:

You have to close the “pop-up” to be able to view another event.

I was amazed at how many stories there were, and this isn’t all, and how crowded the screen became once I loaded all the stories into the XML file.

Anyway I thought I would share it, a slightly different take on presenting on-going information. Oh, I stopped at the 15th (in the hope it was finally some good news, and the fact you have stop somewhere).

Anyway let me know your thoughts, it is just an idea I was playing with. I plan to use it else where to represent different data over time (watch this space).

There are a couple of warnings with this, a, it is 1000 by 800 in size (so viewing it on a netbook doesn’t really work, as I have discovered), b, it might be slow to load because the server I have placed it on is quite a cheap one, c, it is only a prototype of an idea.

Does Oracle and the JavaFX team have any interest in taking on Silverlight’s PivotViewer?

July 1, 2010

I do hope so!!

I read today about Pivotviewer via a PCWorld article, and endured Larry Larsen’s talk / interview. It is a very dull talk, but the software appears to be brilliant! There is a better overview here

Who wants to employ me to write something better in JavaFx 🙂

Being serious though, this is something Oracle should do with JavaFX. Surely a JavaFX data visualisation tool interfacing with a Oracle database is a better model than a Silverlight data visualisation tool interfacing with an Oracle database?

Wow, a Silverlight post on a JavaFX / Java blog? I need to go and lie down to rest my weary head… I have to admit, I do like Pivotviewer…

Tutorial on how to use JavaFX 1.3 with the Twitter API

June 7, 2010

I thought I would put together a basic tutorial, showing JavaFX in action with the Twitter API (utilising Twitter4J). The app will resolve your Twitter followers, show your timeline, and send a tweet.

In the first instance you need to download the latest Twitter4J library. I use NetBeans and so I added Twitter4J to the new project I created for this tutorial. Under the properties, right click, select properties, then select library. Then add the jar file you downloaded.

Ok, with that done, we can start coding. First off we want to create a Java class file, lets call it MyTwitterReader:

public class MyTwitterReader
{
    private Twitter myTwitter;
    private String timeLine;
    private String followersURL;
    private IDs myFollowerIds;

    public MyTwitterReader()
    {
        myTwitter = new Twitter("your Twitter username","your Twitter password");
    }

    public String getTimeline()
    {
        try
        {
            List myTimeline = myTwitter.getUserTimeline();
            for (Status status : myTimeline)
            {
                timeLine = timeLine + status.getText() + '\n';
            }

        }
        catch (Exception e)
        {
            System.err.println("Error! "+e.toString());
        }
        finally
        {
            return timeLine;
        }
    }

    public String getFollowersURLs()
    {
        followersURL = new String();
        try
        {
            myFollowerIds = myTwitter.getFollowersIDs();
            int[] t = myFollowerIds.getIDs();
            for (int i = 0; i < t.length; i++)
            {
                String id = String.valueOf(t[i]);
                User tu = myTwitter.showUser(id);
                followersURL = followersURL + tu.getProfileImageURL() + '\n';
            }
        }
        catch (Exception e)
        {
            System.err.println(&quot;Error! &quot;+e.toString());
        }

        finally
        {
            return followersURL;
        }
    }

    public void sendMessage(String tweet)
    {
        try
        {
            myTwitter.updateStatus(tweet);
        }
        catch(Exception e)
        {
            System.err.println("Error! Failed to send "+e.toString());
        }
    }

}

The first thing to note is we are hard-coding the username and password within the constructor, if you want you can write a new constructor which takes the username and password which have been provided by the UI (but that isn’t for this tutorial). I have also created two methods, getTimeline and getFollowersURLs. These will be used to populate the relevant lists in the UI. Further to this, there is the method sendMessage(tweet) which will be called when we want to Tweet to the rest of the world!

The Twitter object, myTwitter is used to gain access to the Twitter data. In the case of getTimeline, we call the getUserTimeline() method which returns a List of Status objects. I then go through each of these and write to a String object and pass this back (in our case JavaFX). The method getFollowersURLs() calls getFollowersIDs(), which returns an IDs object, I then put this into an array of int, and loop through each value converting the value to a String and passing this to showUser method. This method returns a Twitter User object, from which I can ascertain the Profile Image URL using the method getProfileImageURL() on each Twitter User object.

You still with me? Cool.

Let moves onto the “exciting” bit, the user interface. First things first, I am not a designer and this is a simple tutorial so bear with me on the “look and feel”, of course you can do what you want 🙂

Ok, so the UI will contain the Timeline, your followers pictures, and a input box in which you can type your tweet, oh and a button to send the tweet, this is how I have laid it out:

Ok, so a quick overview of the UI.

  1. You will notice the Tweet button is “greyed” out, this will change once some data is entered.
  2. The Characters remaining will count down with each character we enter.
  3. Both My Followers and My Tweets are empty (well we haven’t coded it all up yet!)
  4. Do you like my choice of colours, very original!

I hope this is ok, now over to the code. In the first instance, I like to encapsulate this in a class rather than have it all in the Main.fx file. So we will create a JavaFX class instance, call it TwitterOverview, below is the code:

public class TwitterOverview extends CustomNode
{
    package var processTweet : function(s: String):Void;
    package var timeLine: String;
    package var followersURL: String;

    var tweet : String;
    var items : String[];
    var followerURLs: String[];

    var tweetInput : TextBox = TextBox
    {
       //styleClass: "twitterOverviewTweetInputText"
       columns: 40
       selectOnFocus: true
       translateY: 40;
       translateX: 120;
    }

    var tweetLength : Integer = bind (140 - tweet.length());

    def tweetButton : Button = Button
    {
       //styleClass: "twitterOverviewTweetButton"
       translateY: 38;
       translateX: 450;
       text: "Tweet";
       disable: bind if (tweetInput.rawText.length() < 140 and tweetInput.rawText.length() > 0) false else true;
       action: function()
       {
          processTweet(tweetInput.text);
          tweetInput.text = "";
          tweet = "";
       }
    }

    package var twitterPersonalTimelineLV : ListView = ListView
    {
       //styleClass: "TwitterOverviewTimeLine"
       translateX: 40;
       translateY: 120;
       layoutInfo: LayoutInfo
       {
          height: 450;
          width: 450;
       }
    }

    package var myFollowers : ListView = ListView
    {
       //styleClass: "TwitterOverviewMyFollowersList"
       translateX: 510;
       translateY: 90;
       layoutInfo: LayoutInfo
       {
          height: 350;
          width: 60;
          margin: Insets
          {
             top: 50
             bottom: 50
             left: 50
             right: 50
          }
       }
    }

    function setup():Void
    {
       items = timeLine.split('\n');
       followerURLs = followersURL.split('\n');

       if (twitterPersonalTimelineLV.items.size() > 0)
       {
          delete twitterPersonalTimelineLV.items;
       }

       for (i in [1..<items.size()])
       {
          insert items[i] into twitterPersonalTimelineLV.items;
       }

       if (myFollowers.items.size() == 0)
       {
          for (i in [0..<if (followerURLs.size() < 10) followerURLs.size() else 10])
          {
              insert
              ImageView
              {
                  fitHeight: 50;
                  fitWidth: 50;
                  image: Image { url: followerURLs[i];}
              }
              into myFollowers.items;
          }
       }
    }

    var screen2 : Group = Group
    {
       content:
       [
          Label
          {
             //styleClass: "TwitterOverviewMyTweets"
             text: "My tweets:"
             translateX: 40;
             translateY: 100;
          },
          twitterPersonalTimelineLV,
          Label
          {
             //styleClass: "TwitterOverviewMyFollowers"
             text: "My Followers:"
             translateX: 510;
             translateY: 70;
          },
          myFollowers,
          Label
          {
             //styleClass: "TwitterOverviewSendTweet"
             text: "Send a Tweet"
             translateX: 30;
             translateY: 40;
          },
          tweetInput,
          tweetButton,
          Label
          {
             //styleClass: "TwitterOverviewCharsRemaining";
             text: bind "Characters remainding {(140-tweetInput.text.length())}"
             translateX: 340;
             translateY: 70;
          }
       ]
    }

    override function create():Group
    {
       setup();
       return screen2
    }
 }

Ok, a few comments to make about this code:

  1. You will notice the commented lines styleClass – This is for the CSS styling. I won’t cover this here.
  2. If you look at the tweetButton code, you will see it is disabled = true if the text within the TextBox (tweetInput) is 0 or greater than 139. This means the button is enabled when you start typing.
  3. The same applies to the text counting down the number of characters, notice we are using rawText.length() to ascertain this.
  4. I hold my hands up to using X / Y positioning, rather than VBox, HBox etc, I know Amy Fowler won’t be happy! 🙂 It was just what I did at the time, and I am being lazy by not changing it. Apologies!
  5. The only other thing worth commenting on is, I have the function to process sending a tweet in Main.fx (we shall move onto that next).

Now it gets exciting…. We move onto Main.fx. As we laid out the UI in the TwitterOverview class, the main.fx is clean and tidy (just the way I like it):


       var fxTwitter: MyTwitterReader = new MyTwitterReader();

       function processTweet(tweet: String):Void
       {
            fxTwitter.sendMessage(tweet);
       }

       var myTwitterOverview : TwitterOverview = TwitterOverview{processTweet: processTweet; followersURL: bind fxTwitter.getFollowersURLS(); timeLine: bind fxTwitter.getTimeline();};

       Stage 
       {
             title: "Application title"
             scene: Scene
             {
                    width: 600
                    height: 600
                    content: 
                    [
                          myTwitterOverview
                    ]
             }
       }

That’s it!!! Fire it up and send me a Tweet 🙂 (@SmeeZee)

Going forward, you might want to play with CSS styling, or perhaps explore the additional functionality available through Twitter4J. Perhaps you want the Timeline to update every X seconds? Have a play, let me know what you do. Of course my style is evident in this code, if it goes against yours I am sure you can get my drift and can code up your own JavaFX Twitter client.

Ha fun.