100 Warm Tunas 2016

Last year I predicted the top 3 results in order in Triple J’s hottest 100. This year I’m back at it again, however, now with a webpage and a Spotify playlist.

Results are collected, optimised, and processed multiple times per day. Instagram images tagged with #hottest100 and a few others are included for counting.

Happy voting!

You can read about the process last year here. However, vote collection is a fair bit more accurate this year.


Press:

Other Mentions:


Edit: Woohoo, the Spotify playlist now has just over 1200 followers, and the website has had over 30,000 hits! That’s massive, thanks everyone!

Understanding and Tweaking some GPX data

As a casual bike rider, I enjoy tracking my rides with Strava so I can take a look at how my ride went and how well I performed throughout.

However, very rarely the Strava tracking application randomly crashes, or gets killed by iOS on my phone, during the ride. This means that the data was never recorded between the point at which the app died and the point when I became aware the app had died.

If we plot this type of failure, it looks something like this:

Map with missing data

Fortunately in this case, there wasn’t too much missing data. However, I was still determined to learn about the GPX format and see if I could patch up the GPX file programatically.

In the specific case of the above map, I was riding north west, and at a point Strava crashed. Between this point and when I pulled out my phone to check my progress, no points were plotted. Google maps interprets this lack of data as a straight line between to the 2 points (as per GPX specification).

If we crack open the GPX file and take a look, we can see exactly what this looks like:

...
<trkpt lat="-33.9014420" lon="151.1066810">
    <ele>6.6</ele>
    <time>2016-10-28T23:38:50Z</time>
</trkpt>
<trkpt lat="-33.8802920" lon="151.0702190">
    <ele>20.9</ele>
    <time>2016-10-28T23:51:14Z</time>
</trkpt>
...

In it’s simplest form, a GPX file is an XML document that contains a sequence of GPS points (with associated metadata like elevation, and other depending on the tracker). This makes it reasonably simple for us to get our hands dirty and begin fixing the data set.

In order to add the missing data back into the GPX file, we need 3 things:

  • The last coordinate recorded before the app crashed
  • The coordinate when the app was revived
  • A list of points of the track we want to use for our data points.

Fortunately, I was able to obtain a list of coordinates for the missing data since I travelled the same path on the return journey (As can be seen on the map above).

The other 2 app state points of interest are reasonably easy to find - just find 2 data points that have a (reasonably) large time distance between them.

In order to process the data, I used a python library called gpxpy which provided some good utilities for reading and processing a GPX file.

With this library, I was able to find the crash point, the revival point, and the list of the points of the track. With this data, I interpolated the start/end times of the crash points onto the track data, and spliced it back into the dataset.

After exporting the data set, we achieve a map that looks like:

Map with resolved data

Quite clearly, this has a few limitations, for example, the calculated velocity through all of the data points is simply an average. However, this did provide me with an improved dataset which I could re-upload to Strava.

You can find all the source for this script on my github

Accurately Predicting Triple J's Hottest 100 of 2015

In 2014, a prediction was accurately made for the Hottest 100 of 2013. The results were posted on warmest100.com.au.

The author of the prediction in 2014 managed to acquire accurate results because Triple J featured a social share button on their voting page, which posted your votes to your Facebook in text form. The author scraped results from public Facebook posts and aggregated all the votes. They managed to obtain 1.3% (1779 entries) of the expected total vote.

Consequently, voting for the Hottest 100 2014 and 2015 did not contain such a feature. Fortunately, voters still felt the need to share these results with their friends, and taking a screen shot or a photo of their screen and posting to social media was a concrete alternative. Using these images posted to Instagram, I was able to accurately predict the results of Triple J’s Hottest 100 of 2015.

Some Cool Stats Before You Continue

  • Triple J Tallied 2094350 Votes (209435 Entries) for Hottest 100 2015
  • I collected a sample size of ~2.5% of all entries
    • 7191 images initially collected
    • I categorised 5529 images as votes
    • ~4900 images contained the words “vote/votes/voting”
  • My Top 3 Results were 100% accurate

You’ll probably find this article interesting, but if you’re super eager, you can Skip To The Results.

Taking Advantage of Social Media

I decided to only target votes that were posted to Instagram, since a high majority of the pictures hashtagged with #hottest100 were in fact votes, and there was a reasonably high volume of them, and most publicly accessible.

I required means to acquire all pictures that had been posted to Instagram. Instagram have an official API, however you are required to have your API app usage approved before it can interface with non-sandbox users. Additionally, Instagram impose a rate limit on non-approved apps, as well as approved apps. I did not have time to waste, and wanted results immediately, so I found an alternative.

Fortunately, Instagram exposes a non-public API through their website ajax loading when you browse to a hashtag. By imitating the web browser with a simple python script using the requests library I managed to download all images from the latest until a cut off date that I specified (the day voting opened).

After scraping the hashtag #hottest100, I expanded my search to #hottest1002015 and #triplejhottest100.

Processing Images

After downloading 7191 images from Instagram, I needed to find an accurate way to filter out the images that were not votes.

I’ve had previous experience with using PIL in Python, so using PIL, I wrote a simple script to sort the photos into 2 categories; photos that appeared white-ish, and photos that were not.

A good vote looked like this:

A Good Vote

Unfortunately, not every image ended up in the right folder, and I ended up with both false negatives and false positives, however I wasn’t too concerned about false positives, as my OCR processing step would exclude them. Instead, I was more concerned about false negatives.

As the image processing and sorting continued, I manually moved false negatives to the positives folder. I calculated about 5% of the non-matching photos were incorrectly classified, however this was due to them being pictures taken of computer screens, similar to the photo below:

A Bad Vote

Some image statistics:

  • 7191 images collected initially
  • 1662 images categorised as non-votes
  • 5529 images categorised as votes
  • ~4900 images contained the words vote/votes/voting

Improving OCR Performance

After experimenting on raw photos from Instagram, I found that OCR accuracy was not very accurate. To remediate this, I utilised Imagemagick to flatten image definition to improve text results.

An improved image

Bringing in Tesseract (OCR)

After weeding out the junk, I still needed to turn these images into readable text.

Using Google’s Tesseract library, I slowly processed all the images and extracted the text from them.

Unfortunately, due to the layout of the Hottest 100 voting website the two columns were broken up inconsistently over the results.

Some were processed as:

Line by Line processing
...
Flight Facilities
Hayden James
Hermilude
Major Lazer
RUFUS
Weeknd, The
ZHU x Skrillex x THEY.
Jarryd James
Disclosure
Kendrick Lamar
Heart Attack {FL Owl Eyes)
(Radio Edit)
Something About You
The Buzz (Ft. Malaya/Young
Tapz}
Lean On (Ft. Mé/DJ Snake}
Innerbloom
...

And others processed as:

Song/Artist line by line
...
Lucky Luke 1 Day
Mosquito Coast Call My Name
Tn ka Right By You
Tuka L.D.T.E.
Half Moon Run Trust
Spring King City
Tame Impala Let It Happen
Saskwatch I‘ll Be Fine
Jungle Giants. T Kooky Eyes
he
...

And others just did not process at all, due to resolution, colour, skewing, or simply because they were a photo of a computer screen:

Bad Image
'VHotllne Bling
Regardless (Ft. Julia Stone)

Parsing the Results

I processed the results line by line, and call these “terms”. These such terms could contain a single song title, a single artist, an artist name with song name, or just junk overhang from a previous line. Initially there were 31062 uncategorised terms.

I processed each term and aggregated number of results for each. This worked really well for songs with short names that were less prone to error, such as Hoops, however did not correctly capture terms where artist name and song name occurred on the same line, or where the OCR library interpreted a few characters incorrectly.

OCR Inaccuracy & Levenshtein

Even with photo enhancements, the OCR accuracy was somewhat subpar for some votes. Some l’s were interpreted as t’s, i’s as l’s, etc. Additionally, the longer the name of the song, the more prone to error it was.

Fiesh Without Blood
L D R U Keepmo Score Fl Pavqe IV)
Yam: unpala The Les I Knew The Bauer
The Tlouble Wilh US

A technique that can be used to fix these spelling errors of single/multi character errors is the Levenshtein algorithm for edit distance. Using this algorithm, we can compare 2 strings and determine how many edits need to be made to make the strings equal each other.

In order to perform this kind of matching, we needed an accurate list of songs that were released this year, along with a list of artists that released music this year.

Using Spotify To Help

To acquire an accurate list of songs released this year, I used Spotify and crawled various playlists from 2015. These included Spotify Charts, Triple J Hitlist, and various other genre-alike playlists.

In the end I ended up with a songs list with 1781 songs, and an artists list with 1229 artists. After the Hottest 100 aired, I compared the results of the countdown to the songs found in my list, and only 6 songs that occurred in the hottest 100 were not in my “truth” list.

During list gathering, I made sure to convert all unicode characters to their ASCII counterparts, so that characters with accents and similar would be matched correctly.

Continuing Processing

Now carrying reasonably accurate artists and songs lists we continue categorisation and processing. The processing algorithm worked in the following way:

  1. Load all terms from every image’s .txt OCR result. Every line is a “term”.
  2. Clean all the terms by turning them into lowercase and stripping whitespace.
  3. Loop through each term:
    1. If term exists in our known songs list, move the term to the songs aggregation and count the votes.
    2. If term exists in our known artists list, move the term to the artists aggregation and count the votes.
    3. If couldn’t find it in either of those:
      1. Loop through all artists in our artist known artist list.
        1. Check if the term starts with the current artist. If it does split it into artist and unknown term. Add the votes to the artist aggregation.
        2. If matched artist, check if the new unknown term exists in the songs list, if it does, add it to the songs aggregation. If not, add it back to the unknown. break loop.
      2. If it didn’t have a prefixed artist, just add it back to the unknown terms.

At this stage, we have a reasonably accurate aggregation of results. We have not yet used Levenshtein string matching. We now have 27294 uncategorised terms, down from 31062 uncategorised terms. So far our results:

==       Results       ==
1   Hoops                          998
2   King Kunta                     765
3   Lean On                        750
4   The Buzz                       646
5   Like Soda                      568
6   Never Be                       484
7   Let It Happen                  476
8   Magnets                        465
9   Do You Remember                409
10  Ocean Drive                    405
==    853 unique terms    ==

==       Top Unknown Terms       ==
1   Your Hottest 100 Votes:        2279
2   Your Votes                     2127
3   }                              320
4   Hottest Io                     248
5   V                              231
6   Throne                         222
7   Triple J?                      209
8   D] Snake                       203
9   The Less | Know The Better     203
10  Asap Rocky                     199
==    27294 unique terms    ==

However, we still haven’t aggregated any votes that had spelling errors due to OCR inaccuracies.

Employing the Levenshtein algorithm, we continue to process the unknown terms. I configure matching to allow lenience based on the length of the term - the maximum edits that were allowed was 2/5 * length of term. The process continues:

  1. For all unknown terms:
    1. Check term length > 3. Break if <= 3. Can’t match a short string.
    2. Match Songs:
      1. Loop through all songs in known songs list:
        1. Compare current song to current term. Get edit distance.
        2. If edit distance == 1, move votes for this term to the guessed song in our songs aggregation, then continue to the next term.
        3. Add distance to a dictionary of value/distances
      2. Using our value/distances dictionary, find the closest match that satisfies our 2/5 * len(term) rule. If it matches, move the votes for this term to the guessed song in our songs aggregation, then continue to the next term.
    3. Match Artists using the same method.

Some of the results of string matching, providing some reasonably accurate re-matching.

[A] weekncl, the -> weeknd, the with distance 2
[A] mm m. -> ms mr with distance 2
[S] km; kunta -> king kunta with distance 3
[A] macklelllore ex ryan lewis -> macklemore & ryan lewis with distance 5
[A] eulsch duke) -> deutsch duke with distance 3
[A] bloc pany -> bloc party with distance 2
[S] nommg's forevev -> nothing's forever with distance 5
[S] t he hllns -> the hills with distance 3
[S] emocons -> emoticons with distance 2
[S] better off without you -> better with you with distance 7
[S] - the less | know the better -> the less i know the better with distance 3
[S] vancejoy fire and the fiood -> fire and the flood with distance 10
[S] too much me togglhu -> too much time together with distance 6
[A] of mons-us and m. -> of monsters and men with distance 5
[S] gmek tragedy -> greek tragedy with distance 2
[S] marks to prove 1t -> marks to prove it with distance 1
[A] rlighx facilities -> flight facilities with distance 2
[A] gang 01 youth: -> gang of youths with distance 3
[A] fka lwlgs -> fka twigs with distance 2
[S] hoine bling -> hotline bling with distance 2

After performing this additional processing, I ended up with 18509 uncategorised terms, down from 27294 uncategorised terms.

That means we were able to successfully categorize 8785 terms via the Levenshtein distance algorithm!

==       Results       ==
1   Hoops                          1011
2   King Kunta                     1008
3   Lean On                        793
4   The Buzz                       667
5   Let It Happen                  637
6   Like Soda                      617
7   The Less I Know The Better     602
8   Magnets                        521
9   Never Be                       520
10  The Trouble With Us            501
==    1143 unique terms    ==

==       Top Unknown Terms       ==
1   Your Hottest 100 Votes:        2279
2   }                              320
3   Hottest Io                     248
4   V                              231
5   Throne                         222
6   Triple J?                      209
7   Thanks For Voting!             174
8   Tapz)                          170
9   Suddenly                       155
10  Once                           140
==    18509 unique terms    ==

Quite an improvement, however still not great. Some of the terms there weren’t able to be categorised which caught my attention included:

9   Suddenly                       155
16  Big Jet Plane                  123
17  Heart Attack                   120
18  True Friends                   114
23  Rumour Mill                    107
35  The Less | Know The            76
63  & Chet Faker The Trouble With Us 46

Paying special attention to The Less | Know The, if I were to add it’s sum to our results, it would have placed 4th, however, the results we already have look reasonably accurate.

Final Results

==       Results       ==
1   Hoops                          1011
2   King Kunta                     1008
3   Lean On                        793
4   The Buzz                       667
5   Let It Happen                  637
6   Like Soda                      617
7   The Less I Know The Better     602
8   Magnets                        521
9   Never Be                       520
10  The Trouble With Us            501
11  Do You Remember                480
12  Ocean Drive                    463
13  Can'T Feel My Face             457
14  You Were Right                 444
15  Middle                         423
16  Magnolia                       381
17  Young                          380
18  The Hills                      369
19  Hotline Bling                  356
20  Keeping Score                  321
21  Embracing Me                   319
22  Mountain At My Gates           318
23  Loud Places                    300
24  Run                            298
25  I Know There'S Gonna Be        287
26  Some Minds                     287
27  Say My Name                    283
28  Fire And The Flood             280
29  Visions                        275
30  Greek Tragedy                  274
31  Long Loud Hours                272
32  Shine On                       254
33  Asleep In The Machine          249
34  Leave A Trace                  242
35  Like An Animal                 235
36  Something About You            224
37  Dynamite                       224
38  All My Friends                 218
39  Deception Bay                  217
40  Downtown                       210
41  Ghost                          200
42  Son                            196
43  Hold Me Down                   196
44  No One                         196
45  Kamikaze                       196
46  Puppet Theatre                 192
47  Vice Grip                      191
48  Forces                         185
49  Better                         185
50  Counting Sheep                 184
==    1143 unique terms    ==

Some Notes

  • Run appeared so high on the leaderboard because both Seth Sentry and Alison Wonderland released similar tracks titled RUN/Run. Since I lowercased all comparisons and removed special characters, these votes merged.

Improving the Analysis

After reviewing the method used for analysis, I have identified a few places for improvement that could possibly improve the results.

  1. Improved Levenshtein Algorithm. The Levenshtein algorithm is great for calculating edit distance, however I could not weigh edits of similar characters such as t’s, i’s and l’s less, thus improving matching due to OCR inaccuracies. I expect that string matching could have been significantly improved if this was explored.
  2. Songs that had long titles, such as The Less I Know The Better generally were split across multiple lines. This caused their aggregation to not sum correctly. It would be good if I could determine if a song was split across two lines.
  3. Songs that were in the format of artist song and were spelt incorrectly were most likely not picked up by string matching, as we only matched against songs and artists individually. In order to improve matching for this, an additional list for joined songs/artists could have been used and compared against for remaining terms.

Some Cool Stats

  • Triple J Tallied 2094350 Votes (209435 Entries)
  • I collected a sample size of ~2.5% of all entries
    • I collected 7191 images collected initially
    • I categorised 5529 images as votes
    • ~4900 images contained the words “vote/votes/voting”
  • My Top 3 Results were 100% accurate

Sprinkle

Here’s a quick demo of something I quickly jammed together over the weekend for my Dad. More info to come, along with additional pictures, circuitry, and some proper screenshots

Basically it’s an iOS app to control solenoid valves via a Raspberry Pi over a JSONRPC interface.

CSESoc Hackathon

A couple of weeks back, the society I am a member of at Uni hosted a hackthon event, sponsered by Freelancer. For the uninitiated, a hackathon is an event where programmers literally turn pizza and drink into applications/code. (But in all seriousness, it’s an event where programmers develop a cool idea in a small timeframe and compete to be the ‘best’ product).

I formed a team with 2 friends from Uni. We set out to build a web platform for students of UNSW to list projects they have worked on in an easy to use web directory that they could use for employment and their own portfolio.

The webapp is written in Python/Python-Flask, uses MySQL as the backend (because mongo hates many to many relationships), and use Bootstrap to style the frontend, statically served from the server.

We wanted the following features from the service:

  • A project has:
    • Web URL
    • Download URL
    • Marketing URL
    • Markdown formatted description
    • Ability to upload screenshots of the project
    • A project can have multiple contributors
  • Project Page:
    • Showcase of all projects the user has worked on
    • About me for the user
    • Show who the user follows
    • Show who is following the user
  • Home Page/General:
    • A-Z listing of all projects
    • Show latest 3 projects on the home page “ShowCase”
  • Logins use UNSW’s LDAP service, so it’s all UNSW SSO.

There are some additional features we wish to work into it, such as reading README.md from github projects.

There are a few bugs hanging around still, along with some non-implemented features, such as multi contributors for a project. We’ll eventually get around to these, and finally launch it!

We plan to put it up on http://showc.se/, a domain I purchased for the project. It’s a nice play on words, and also is a valid regular expression, which matches “ShowCase”, but also is a play on CSE - Computer Science and Engineering.

It’s probably important to note that we came first in the Hackathon, each of the team members winning a UE Boom portable bluetooth speaker thanks to Freelancer!

Stick around for more, i’ll update this post when it’s live!

CTF Season

It’s currently CTF season, and as a member of UNSW’s security society, that means I get to play!

We began the season with CSAW CTF, where we (team K17) placed 1st in Australia/10th overall.

I did not participate in this CTF as much as I would have liked to, since I was already pre-occupied with the CSESoc Hackathon, however, I did lend a hand with Web 500 - A fake dating website where the aim was to recover Donald Trump’s TOTP key as well as his password. I managed to solve half of the challenge by finding an SQL injectable endpoint in a CSP reporting endpoint, where I dumped a password hash and other info about the account. We recovered the password hash using a dictionary attack, However the full solution required dumping of source code to determine how the TOTP key was generated, which another member of the team did, and thus solved the challenge.

The following weekend, Trend Micro CTF was running, which K17 also played in. We ended up coming in at 1st place globally out of 359 teams - A fantastic effort. Once again, I only participated lightly in this CTF. I worked on an Android APK reversing challenge, which I solved over the space of 2 hours. I will post a write up of this challenge soon!

Additionally, I was selected to play in CySCA (Australia’s Cyber Security Challenge) for UNSW3. UNSW entered 5 teams. My team (of 4) placed 3rd overall in the competition, but the entire UNSW effort was also amazing:

  • 1st: UNSW1
  • 2nd: UNSW2
  • 3rd: UNSW3
  • 4th: UNSW4
  • 29th: UNSW5

I’ll be posting my write ups over the next few weeks, explaining my solutions to the problems that I solved for these CTF’s!

Lolcommits!

It’s been a while since I last wrote a blog post, I’ve been busy working, writing code, and doing general work for university.

I’ve been scheduled to do a talk for UNSW’s CSESoc about git, which has given me great motivation to go and find cool and awesome things to do with git. I found Lolcommits! See the left hand menu of this page (on desktop only), you can see my latest commit message (and a funny photo, hopefully) for projects I have enabled lolcommits on.

My idea stemmed from me joking about how it would be funny if it automatically uploaded the image to my blog, but slowly evolved into actually becoming a thing.

All that’s left now to do is for the latest image to be automatically uploaded after it is created!

A Brilliant Explanation of PID

PlaylistGrabber 1.1 (UI Revisions)

Due to the large response to the initial release of PlaylistGrabber, I have quickly revised some of the UI and functionality to bring it up to scratch with user expectations.

Changelog:

  • Added App Icon (Green iTunes yeah close enough)
  • You can now select a playlist by clicking the entire row, not just the checkbox
  • PlaylistGrabber now remembers what playlists you selected last time
  • You can save your progress of playlist selection by pressing “Save Selected”
  • PlaylistGrabber now remembers what XML File you last used. Quick and Easy Startup.
  • PlaylistGrabber skips copying files that already exist in the destination (good if you’re using a cloud service). It will however re-generate M3U Files so you will get playlist updates.
  • Tableview now has pretty icons
  • Tableview nests items inside folders at their level as depicted within itunes. Wish i could indent the icon too.
  • “About” window updated.
  • Auto build incrementing (current release is build 110)
  • Automatically quits app on window close.

Things I’d Like it to do better

  • Nest icons for folder indentation levels
  • Delete songs when a playlist is de-selected. This could prove quite tricky

I’m pretty happy with what I have achieved over the few hours I’ve worked on this project. And after all, I’ve learnt how to program for Mac OSX.

Download

You can download release 1.1 build 110 here:

  • PlaylistGrabber (.app, drag to your applications folder to install, run whenever you want to enumerate music on your device)

You can obtain the source and self-compile here.

PlaylistGrabber for OSX/iTunes

Spotify was great. I had my music everywhere, could add new music without a computer, however it lacked in a major area – Playlists. It seems to be a growing trend of music players to suck at this. Being unable to shuffle all music on the device is also a massive drawback. I will be cancelling my Spotify subscription once my 3 month trial is over.

Let me introduce PlaylistGrabber for OSX. This is my first Cocoa application, targeted at 10.8 and upwards (not tested much). PlaylistGrabber reads your iTunes library XML file, and allows you to choose playlists to export. It creates a folder structure that you can drag and drop onto your device (or export directly to the device if you have mass storage capabilities). It exports playlists in M3U format and understands that the duplicate songs in different playlists are the SAME song – so no stupid duplicates in your library, just as iTunes handles it.

Due to the nature and simplicity of M3U playlists, most music players understand these, including PowerApp, Samsung Music App and Google Play Music. This is good news, as now you are free to roam to other solutions than DoubleTwist for all your iTunes Syncing needs.

Eventually I will tidy up the application, however at this present time, I do not have enough time to do so. Eventually I would like to make the app do the following:

  • Save Chosen Playlist Preferences for re-loading later on if a user decides they want to re-sync Implemented in newest version.
  • Sync Daemon – Watches when Library Changes, and writes changes to a sync directory, from where you could auto sync with google drive
  • Wifi Sync (With a client app on the phone)
  • Better Async Handling so the program doesn’t appear to “Lock Up”

If you want to download this and try it out, feel free to: PlaylistGrabber. If you come across any bugs or have any suggestions, let me know, it’ll be nice to track them in the future for new releases in summer this year.

Latest version is available here

Flask, Flask-SQLAlchemy & Flask-Security

My last 3 weeks has mostly consisted of programming for a Flask Web Environment.

What is flask? Flask is a fancy python library allowing rapid web application, api, and interface development. It tackles many of the hard parts of programming web apps in nice and easy paradigms.

Flask has many extensions available, two being Flask-SQLAlchemy and Flask-Security. These two are must haves for any kind of application development involving a database and a level of security management.

Back in my PHP days, I would slave away and create a fancy PHP permissions structure for whatever web application I was writing. Horrible. Probably filled with vulnerabilities & all sorts of bad things. Flask-Security is made so you don’t need to do this.

How about SQLAlchemy? SQLAlchemy is a database interface wrapper, but it’s more than just a wrapper. It does ORM, which means you can represent your database entities as objects/classes. A very similar paradigm to both CoreData on iOS AND my own, NWRestful/NWManagedObjects framework.

So, how do we use all this?

As this isn’t a tutorial, I won’t go through it, however there are some AMAZING examples hosted on the Flask website and the Flask-SQLAlchemy website. I cannot say the same for Flask-Security however.

Over my first project, programming for Flask/Flask-security, I found the security library’s documentation to be greatly lacking. This was a slight drawback, however once I got my head around the library, everything went nicely.

I will aim to upload a Flask example package/boilerplate with the structure of how I use flask within the next few weeks.

Bridging the Gap: iTunes to Android

I hate proprietary things. Specifically iTunes. It’s a great music manager, it’s robust, stable, and has a brilliant store. It also offers fantastic integration with iOS devices. As much as that is great for iOS users, it sucks for anyone on Android. In my opinion this is driving people to move to streaming music services such as Google Play Music or Spotify (both of which I am contemplating).

There are various tools available which bridge the gap. Apps such as DoubleTwist for Mac & Android do the job, however they’re just too clunky.

For me, leaving my iTunes Library and picking up some new music management tool would be a big hassle. iTunes has all my music, including over 300 of my playlists. I need a program to bridge iTunes and my Android Device.

At the moment I’m on the verge of writing a tool to extract iTunes music, track playlists and keep music in sync onto my Android phone. My roadmap for the tool is as follows:

  1. Read iTunes Library & Copy Playlists onto device without duplicates
  2. Synchronise Playlists with the device, so songs removed in iTunes are removed on the Phone.
  3. Write metadata to the device for playlists
  4. OR Write a music playing app – however this may be overkill.

Anyway, this will be a learning process. I’m aware tools are available to do this task, however building one tailored to my needs to possibly the best solution in my case. Looking at the iTunes Library XML files, they look relatively easy to work with, and maybe eventually I’ll set up a auto importer for my “external purchases” to automatically add them to my Library and put them in the right folder on my computer.

Picaxe Microcontroller

This week was my last week for the semester at uni. At our last computing lecture our lecturer told us there would be a micro controller question as part of the test. He did state that if we wrote a micro controller emulator in C and shared it, it would be allowed into the test. This is greatly useful for checking machine code during the exam.

I decided I would set about writing up an emulator. I finished the task with no issues. I’m very happy with my emulator (of a fictional microprocessor).

The microprocessor is noted as the “8005” chip. A chip with 256 bytes of memory. It’s got about 17 different instructions. The sorts of programs we have written in machine code for these are amazing, we have written halving functions, as well as wondrous number generators. I’m certainly amazed.

Now, whilst writing the emulator, I remembered I had a picaxe micro controller stored away in my cupboard. I brought it down after I was done and started playing. My interest in this also comes about due to my father wanting to read RS232 data off our solar panel inverter and viewing it on his iPad. This is my challenge tomorrow – this was just the warm up.

I hooked up the picaxe on a breadboard, and wrote a basic program. However it’s made me surprisingly happy. I can control a LED via serial communication from my computer.

iOS7 Remote Notification Removal

I’ve noticed since the early builds of iOS7 that notification centre on both the lock screen and within the phone, now has this really nice feature.

It appears as though specific services such as Google Plus and Facebook which distribute notification to users phones will now remove the notifications from the lock screen if they have been viewed somewhere else (ie, on a computer).

A quick google search of this reveals no specific results for this phenomenon. I’m 100% sure this is a new addition to the OS, as previous builds of iOS with Facebook notifications would not be removed from your lock screen.

At the present time I am unsure of how the implementation works, however, I would assume that it would involve APNS sending a badge count of 0 to the phone, ideally removing all notifications. I will be testing this after my HSC and determining it’s results.

PDO and Sanitisation

No doubt in the world of programming, sanitisation is one of the most important things when implementing a web database of some sort.

Typically I’ve used mysqli_real_escape_string, which does the job, but it’s so tedious to type. I only recently found out about PDO, and prepared statements.

Due to my current decision to use SQLite instead of mysqli, I’ve needed to use PDO to escape all of my input data. My input data had double quotes, which is not escaped by the standard SQLite escaper. The SQLite docs state not to use addslashes to escape the strings either.

After googling for a bit, I realised my only option was to use PDO – but to my surprise it was extremely simple. If anything just as easy as writing a standard query. The only modifications needed was my variable insert structure in my managedobjectmodel super class.

I’ve now got to work on upgrading models inside the database (although I have a basic version of this already working), until I release the code on github.

Anyways, what I’m trying to say here is – don’t risk SQL injection, use PDO, it may take an extra half minute to write, but it’s worth it in the long run. Alternatively, use my framework or similar to do the heavy lifting for you.

Rest, Rest, and more Rest

Nope, I’m not talking about sleeping. I’m talking about quite the opposite. Actually, as I type this, I should be sleeping, however, I am not. To be completely honest, REST is taking away my own rest.

Okay that’s enough late night rambling – Here’s what’s happening.

I’ve continue to play and implement with this REST server I’ve written, and now it’s time to play with the actual client data retrieval. In this case, the iPhone. It took about 3 hours for me to get my head around RESTKit for iOS, but with the help of this tutorial, I managed to get it all set up, and working nicely, including integration with coreData for persistence.

Right now i’m up to the position of implementing authentication, however, I’m split between how I should model the getting of objects. I’m considering writing my own Objective C class to handle all the sending and retrieving of data – It will still interface through RestKit, but it’ll hold all the methods and keep everything neat and tidy – something I REALLY need to do with this project, as all code will be submitted as my major work for SDD.

Anyways, that’s probably enough for tonight, I’ll take another look at it in the morning, and maybe write down some plans for the class. It’s now real REST time….

The magic inside NWRestful

For both my own future reference, and for others, I’m going to write some basic ways to use NWRestful – well, the data structure framework underlying it.

Creating a Model

To begin with, you’re going to need to create a model a data structure. This is relatively simple:

<?php
	class EventModel extends NWManagedObjectModel {
		public $name;
		public $place;
		function __construct() {
			parent::__construct();
			settype($this->name, 'string');
			settype($this->place, 'string');
		}
	}

?>

There’s not too much to it.  Simply define your variables, and define their data types. You must ensure you append the word “Model” to your class name. (Make sure you import the “NWManagedObjectModel” class if you’re not using an auto importer).  Once you’ve done that, we will see where the magic lies.

Creating & Saving

NWManagedObjectModel does most of the work here. Upon the first save of a new object, it sets up the database, creating the columns and setting their data type. Let’s see how we create and save things:

<?php
require_once("EventModel.php");
$event = new EventModel();
$event->name = "Some Name";
$event->place = "Some Place";
$event->save();

?>

Pretty simple right? We just saved that object to the database – And don’t worry. Arrays and sub-objects will be serialised and encoded before insert, and will be exactly the same as you saved them upon retrieval.

Retrieval

This is where we need to use a context class. We could ideally query the DB directly, but the context will give us back the objects as the object we saved into the database. Easy right?

require_once("NWManagedObjectContext");
$context = new NWManagedObjectContext();

//Option 1. Query all objects
$events = $context->getEntitiesForStatement("Event");

//Option 2. Query using a WHERE statement 
// nb. You need to filter your own injections. 
$events = $context->getEntitiesForStatement("Event", "`place` = 'Somewhere'");

//Option 3. Query for a specific instance.
// (Objects are given a unique instance ID upon insertion/creation).
$events = $context->getEntityForInstance("Event", 22);

Now, if we try view the $events variable, we should see:

Array(
    [0] => EventModel Object {
        place->"Some place"
        name->"Some Name"
    }
)

Everything is retrieved as restored into an EventModel object! That’s easy! Need to make a change? Just change the variable:

Modification

$events = $context->getEntityForInstance("Event", 22);
$event = $events[0]; //Grab the first result.
$event->place = "Some new place";
$event->save();

It’s as easy as that.

Wrapping Up

By the end of next week, I plan on releasing my DBMS/Sqlite interface as a separate package, as this really is the feature here – the REST library really is nothing. I Hope to see this framework get adopted in the future by many others. I do plan expanding the framework to more DB systems, such as MySQL, however, SQLite is keeping it simple for me at the moment in my current project.

NWRestful – PHP REST framework

Well, between my earlier blogging of photos and now I’ve been writing a little PHP REST framework. It has come from my need for a REST interface for my iPhone app for my major project. After much searching, I was unable to find a decent REST framework, or at least something that was easy to understand and use.

I began to follow a tutorial I found here, which served as the complete base of the project.  That blog post doesn’t show you much, only a bunch of simple code snippets from a working server. I put some of these to use, writing my own implementations. The one thing this tutorial was missing was working with models – but that’s fine, the tutorial was just about REST.

I began wondering about CoreData on iOS and the magical ways it does things to data. It transforms the data and uses the NSManagedObject class along with a NSManagedContext to do all the database magic. I thought to myself – “why not take this approach to the issue”. So that’s what I did exactly. Now, I had multiple DB options, but I opted to use SQLite3 simply because I could take the DB around with me using git.

I wrote a standard class for handing all the data objects called NWManagedObjectModel. This model holds the methods for saving and purging. Any new data model can be created off this class. The subclass can define it’s own purging methods, saving methods, or whatever else it needs. The main idea is that each model simply hold the variables and their types.

When an object is saved, an SQL query checks to see if a table has been created, and if it hasn’t will create one. The table will be created to identically replicate the PHP object model/class object type you are saving. Once the table is created, it now inserts/updates the object to the database as  a new row.

Creating objects and adding them to the DB is relatively simple – Simply create a new PHP object for the model/entity you want, and call the save() method on the object.

Retrieving objects is equally easy – Simply create a NWManagedObjectContext object, and call either getEntitiesForStatement($entity, $statement), calling your own custom SQL statement, OR getEntityForInstance($entity, $instanceID). These return an array full of the entities/php objects you requested.

As for the REST and how it handles these different object model types – It’s really transparent. All you need to do is define a controller for the model, you don’t even need to define any methods on the class (provided you subclass “RoutingController’), all the work is done within the RoutingController class.

Anyways, that’s probably enough late night rambling – plus it’s a school night.

Here’s a link to the GitHub Repo. Currently i’m licensing under Creative Commons Attribution-NonCommercial 3.0 Unported License

4 Monitors, 1 GTX660TI

The other day my VESA wall mount arrived in the mail. It’s a simple $16AUD wall mount, able to hold about 10KG. I drilled the holes in my wall, and hung up my 4th monitor – another acer, except it’s only 20″ (V203H).

I managed to get all four monitors to run from my GTX660TI inside OSX (finally) by installing a nvidia driver for CUDA (which I guess comes with a display driver). 2 monitors are running from DVI, One from HDMI to DVI and another using DisplayPort to DVI.

Weekly Roundup 27 May 2013

I totally forgot yesterday that I had to write this post - however, I’m doing it now, so I guess this kind of counts. This week’s been a bit of a busy one. I’ve had a couple of school assesments to do, as well as a tonne of homework, which is still waiting for me to do, Anyways, let’s get into this.

This week I discovered the world of cocoapods (however almost ruined my whole SDD project git repo, but I rolled back and was all good.), The world of RESTkit, which seems like it’ll solve all my data persistance troubles in my SDD project, and finally all about reaver WPS, which i’ve written a nice little piece about below;

Before I get into the story of BT linux and reaver, I must share my obsession with the Great Gatsby Soundtrack, which caught my eye when I heard the song ‘Together’ by The XX on pandora radio. I highly recommend anyone to check it out, due to it’s gigantic mix of genres.

Continuing on, this week I ventured into the world of reaver-WPS - But this came at a cost - My time. Lots of time. I ran BT linux from a live USB HDD, created with unetbootin, which worked really well - except i couldn’t save anything, but I knew that. So on saturday, my surprisingly cheap GTX660TI came ($140 factory repaired + warrantee), so I decided if I was going to install the CUDA drivers, i’d actually like them to stick around after reboots. I ran the BT installer planning to install onto a 500GB disk I had in my cupboard, however, silly me - I didn’t unplug my stash of HDD’s in my PC after plugging in the new 500GB. Wisely, I did install BT to the correct partition - however, this is where it all went wrong. The bootloader decided it would install to /dev/sda0.. My SSD. Containing my OSX86 bootloader. Somehow the installer broke the GPT, and even after trying about 20 times to re-install many different flavours of the a GUID bootloader, I eventually came to the realisation i’d screwed something up pretty badly. I made a full system image using Carbon Copy Cloner to a sparseimage file (I also have a live recovery partition which is a weekly backup of my system, which is how I planned to restore this sparseimage file. I didn’t want to write over the recovery drive as I was worried i’d broken something inside OSX). Anyways, Booted into recovery, formatted my SSD, and cloned my sparseimage back to it. I installed chimera, and rebooted. Horray! bootloader. 4 hours later, I now have a working osx86 system again. I rebooted again, this time, ensuring all my hdd’s were unplugged. I ran the BT installer, and all good.

I continued to install the Nvidia CUDA drivers, however, I was unable to get pyrit to work on my card. Benching pyrit on my CPU yielded results around 4000pmk/s - it now wonders me what my GTX660TI could be doing. The amusement of BT linux began to wear off, and I decided i’d save it for another rainy day, reaver WPS seemed like a bit of an eh sort of thing anyways - Right now, I needed to sort out the GTX660TI in OSX.

3fba7794afbb11e2ad1322000a9e28e6_7 I have 3 monitors, so how my original setup worked prior to my new 660 card, was pretty fragile. I originally started with a GT640, running a single monitor, however, once I acquired another two monitors for either side of my primary, I realised, not all 3 ports on the GT640 would work at the same time inside OSX (But it works in windows :S). I had to enable the HD4000 integrated to make this work. I needed to find a way to allow these to work side by side, but eventually found a way using a <device property> string inside my com.chameleon.boot.plist file.  This worked nicely - until my 660 came along. I know the device property was for the 640, so that was no use to me, so, I had to ditch it, but all my attempts to make anything work failed. I’m unable yet to determine if I can use more than 2 outputs on the 660 as I don’t have a DP to DVI or a HDMI to DVI cable (I have ordered one however). My attempts to fix this display dilemma, was simply to disable the onboard gfx, and put back in my GT640 (Which i know is also natively supported). My system is happy, and now has really nice CUDA acceleration.

 

Listy: Progress is being made

So, as shown in my previous post, my app ‘Listy’ was recently approved on the app store. The initial release had a bug for all iOS5 users, causing it to crash, but I released version 1.02 to fix this.

Anyways, 2 weeks later (when I should be studying for my HSC IPT/IT), the project has become so appealing to work on. I’m learning so much through this. The latest version I’m working on (Version 1.03), has been modified to work on iPhone as well as iPad, better customisation of lists, it’s quicker, it’s easier to use, and uses CoreData. 

If you take a look at the following screenshots, you can see the new interface elements, such as being able to ‘star’ items on a list. At the present time I’m working on allowing lists to be customised to specific user needs, ie – allowing them to choose what they are sorted by, allowing priorities of items, or to create a traditional to-do list. You can now edit items on the list, and depending on the sort of list you set up initially, you can attach geolocations, longer descriptions, links, and images. You can also choose who gets to see the list, determined by the email address you enter for the user. A much more private way of sharing lists.

As this new version uses CoreData, the syncing methods are going to be a challenge on the server side, especially with shared lists. I’ll work it out, but this next release could be months away.

As for my own personal education – boy have I learnt a lot. I’m still learning ObjC, self taught, just like all my other languages, but as I do more advanced things, I understand the concepts as a whole. For example, the custom star button, I implemented a delegate method to tell the parent controller that the star had been checked and needed to update and save CoreData. I’ve also learnt CoreData, how to open records, and modify – and in turn, this actually allowed me to understand the concept of pointers*. It’s all so much magic. It’s all hard work, but it’s really what I love doing.