This article summarizes my adventures in Twitter data-mining using Paterva’s excellent Maltego product.

If you’re not a Tony Hawk / Skate fan you’ll have to use your imagination of what else could be possible with this product; Trust me, there is a lot you can do here. At the end of this article, I have provided a list of references that I found useful in getting to grips with Maltego and the Twitter API.


Trigger

Recently, Tony Hawk (the world famous skateboard legend) enlisted some of his twitter followers (tweeps) to hide packages containing skateboards and other goodies in cities and towns around the world. He then tweeted clues, which his remaining tweeps used to hunt down the packages…see my previous two posts for an insight into the experiences of both a hider and a seeker.

If, like me, you’re a bit of a geek, you may well have wanted to see a nice Google map showing the location of every package. You might also have wanted to gather some data about who hid packages, who found them, how quickly and possibly see some pictures etc.

Carry on reading if you want to know how I (well, “we” really) did this using Paterva’s fantastic Maltego product, some hideous Perl script and some spreadsheet print outs with a pen and paper.

Note: If you’re a hider/seeker, and you spot something amiss, please tweet me @TheSuggmeister and let me know what I got wrong.

For everyone else, here’s the Google map @MrsSuggmeister and I just completed. (Click on this to get to the Google map, and while we’re at it, click on the other images to see bigger ones)


Here’s some maps  @SweetJerome sent me (more about him later). These don’t show the international hides which were 2 in Hawaii (Maui, Big Island), 1 in Australia (Kingscliff South NSW), 3 in Canada (Whistler, Toronto, Winnipeg), 1 in Alaska (Anchorage), 5 in the UK (Glasgow, Birmingham, Essex, Portsmouth, Basingstoke)




All US Packages Packages Hid on 13th Oct Packages Hid on 14th Oct





Packages Hid on 15th Oct Packages Hid on 16th Oct Packages Hid on 17th Oct


And here’s a table with what we think is the mapping between hiders and seekers.  Tweet/DM me if I got anything wrong (not that’s it’s all that important I guess).


Read on for the geeky bit….

The Mission

Mission: Create a Google map of the locations that #THTH (Tony Hawk Twitter Hunt) packages were hidden in, who hid them, who found them and any pictures associated with the hide and the find…




Hider (@TheSuggmeister ) Hiding Spot Finder (@Steven_Gill )




Google Map Piece of cake I thought.


Before we start….Some things we already knew or at least assumed

#1 : People who hid packages were all ‘friends’ of @HidingIt. I know this because I was one of the lucky chosen ones who got to hide a mystery package (actually, it appears that the friends of @HidingIt were a subset of the hiders. I guess Tony hid some himself).

#2 : This subset of hiders had to tweet @tonyhawk with their location and why they should be trusted. Here’s what I wrote…

#3 : The people who found packages all had instructions to tweet @IFoundOne once they found one.
Note: That doesn’t mean they did. Some people will just never follow instructions eh ;-)

#4: Tony tweeted each find and each of those tweets contained the word ‘Found’ and typically where it was located.

#5 : Finally, everyone was encouraged to use the twitter hashtag #THTH when communicating on the topic.

Side-note: How #THTH never became a trending topic I have no idea. Free very cool stuff signed by a skateboard legend with 1.8million+ tweeps = no trending topic, yet #middleclassreggae = trending topic. “Go figure?” (as they might say in the USofA).

So, we know (sort of)

  • Who hid packages.
  • Where the hiders were located (but not necessarily where they and their package were located on the day of the hide.)
  • Who found packages.
  • What state they found packages in. (Ermmm. Which US state, as opposed to “slightly damaged”).


Hypothesis

My hypothesis was that people who hid packages would tweet the finders of the packages with a note of congratulation.

This hypothesis was based entirely on my own behavior, which the reader should note, is not a good basis for a hypothesis, in this context or otherwise.


Pic #1. General Concept


Initial approach

My initial approach (prior to finding out about Maltego) was to search twitter and cut&paste/scrape until I could complete the following.  Naively, I figured this would take maybe 2 hours or so:

#1. Find the Hiders

  • The people who hid stuff (list of all the people who @HidingIt were following)

  • Find out where they were located based on the tweets they sent @tonyhawk after September 12th 2009 (the date he requested volunteers to hide packages).

#2. Find the Seekers

  • Search for all tweets written by @tonyhawk between Oct 11th and October 20th and search twitter for all mentions of THTH between Oct 11 and October 20th.
  • Filter those tweets for the words “FOUND”  and extract the twitter @usernames.

#3. Map Hiders to Seekers

  • Map hiders to seekers based on their locations
  • Create a google map.
  • Feel sense of accomplishment

Now, it should be noted that:

  • I’m relatively new to twitter and,
  • I previously had no idea about http://search.twitter.com/
  • I was in a bit of a hurry (we were a couple of weeks away from the birth of our little boy)

…so I sent out a quick tweet asking

This came back rapidly from @l0sthighway (who’s a smashing chap BTW, thanks for your help @l0sthighway, I learned stacks)

In the red corner – Introducing Maltego

Take a look at www.paterva.com. Maltego is exactly to sort of tool I love. It’s a visualization geek’s fantasy (if you like GraphViz, you’ll love this).

There’s a community (free) version and a commercial (pay for version). The commercial version is around $430
USD (and $320 each year after), which is a bit to pricey for a geeky side project (distraction) while @MrsSuggmeister and I waited for our baby to arrive.

NOTE: There’s $110 off this Christmas… use the coupon ‘MerryXmasMaltego’ on

http://tinyurl.com/maltegorocks.

The problem I had was that the commercial version seemed to have some functions I really wanted to use (exporting results,fewer restrictions on transforms, that kind of capper). ARGGGHHHH!

I sent a quick email to Paterva and explained what I was trying to accomplish and that a short term license would be great if they could come to some arrangement about $’s. I also followed up with a phone call (and why not). My call was answered by Roelof Temmingh, one of the key authors of the product (and owner of Paterva) who suggested “have you checked your email?”.

Wow!

Like Christmas coming early, he’d sent me a 21 day license to help Mrs. S and myself on our little quest.  Before I ramble on further, I also need to call out the amazing support and collaboration from Roelof and Andrew MacPherson (AndreMohawk), who would have been well within their rights to suggest I “poke off” and “read the manual”.

If you’ve never heard of Maltego before, I’d recommend having a look at this slideset from BlackHat Europe 2008, and in particular slide 18, which introduces the architecture (note, that the Paterva transforms run on their servers, unless you buy your own).


Revised Approach

After a brief look at the Maltego, I revised my initial plan thus :

#1. Find the Hiders (and roughly where they were located)

  • Get a list of all the people who @HidingIt were following (all the Hiders);
  • Get their tweets to find out where they were located based on the tweets they sent @tonyhawk after September 12th 2009 (the date he requested volunteers to hide packages);
  • See who they tweeted using the ‘AffTwitter [This Person Wrote Tweets to]‘ transform;
  • See who they received tweets from using the ‘AffTwitter [This Person Received Tweets from]‘ transform;
  • Convert all results to twitter users, using the ‘To Twitter Affiliation [Convert]‘ transform.

#2. Find the finders (and roughly where they found stuff)

  • Search for all tweets written by @tonyhawk between Oct 11th and October 20th using the ‘AffTwitter [This Person Wrote Tweets to]‘ transform;
  • Search for all tweets written to @tonyhawk between Oct 11th and October 20th using the ‘AffTwitter [This Person Received Tweets from]‘ transform;
  • Search for all mentions of THTH between Oct 11 and October 20th;
  • Convert all results to twitter users, using the ‘To Twitter Affiliation [Convert]‘ transform.

#3. Map hiders to seekers.

  • The steps above should (if things work out per my hypothesis) show a nice mapping of conversations between hiders and seekers.
  • Map hiders to seekers based on their location
  • Create a Google map.
  • Feel sense of accomplishment


Starting Off

Before we can jump in, we need to reach the starting line…

Get a starting entity (known in Maltego as an ‘AffliliationTwitter’) of our target twitter users,  @HidingIt and @tonyhawk

Note: In v2.x of the product, we have to derive these. i.e. We can’t just plonk @HidingIt onto our map and have Maltego figure out that it’s an AffiliationTwitter. I achieved this by  following these steps (if there’s a quicker way, let me know).

Start with a  Phrase, in this case “Sweetjerome”, as I knew the user @SweetJerome tweeted recently and followed @HidingIt ( and not too many others). In other words, I’d quickly be able to get to an AffiliationTwitter of @HidingIt.  You can achieve this by:

  • dragging the Phrase icon from the left hand menu and dropping it
    in your main map window
  • double click where you see the text ‘Phrase’
  • start typing the phrase you wish to search for,
    in this case “Sweetjerome”/

OK? why did I do that? Why didn’t I search for the phrase “HidingIt”?  Ordinarily I could have started with “HidingIt”,  but the twitter user @HidingIt hadn’t tweeted for a long time and is no longer appears to have any tweets indexed by twitter search (as far as I can tell).

Once you have your phrase, right click on it and select the “To Tweets [Search Twitter]” transform (see image below – click the image to see a bigger pic).

Here what we end up with after running the transform. The prickly looking purple icons are ‘Twits’, i.e. the tweets containing our phrase.

We now want to generate some AffiliationTwitter entities (Twitter users). We can achieve this by selecting all the ‘Twits’, right clicking and selecting the ‘To Twitter Affiliation [Convert]‘ transform  (see image below).

Here what we end up with after running the transform. We can clearly see our target twitter user @SweetJerome

We now want to obtain a list of @SweetJerome‘s friends (the people he follows). We can achieve this by right clicking and selecting the ‘To friends of this person’ transform (see image below).

Here what we end up with after running the transform.  Here’s a Maltego map of this stage for you to play with.

We can now see our true target, @HidingIt. To keep things tidy, we can select everything other than @HidingIt and cut it out, leaving just @HidingIt on our map (see image below).

It’s probably worth saving your map at this point as it’s a little bit of effort (a few minutes) to achieve this. Here’s a Maltego Map with @HidingIt for you to play around with.

Like I mentioned at the start of the article, if people have better approaches to deriving an AffiliationTwiiter, I’d be very interested to hear about them.


First True Step

#1. Find the Hiders (and roughly where they were located)

We can achieve this by generating a list of @HidingIt‘s friends with a ‘To friends of this person’ transform.

Before you do that, shift the ‘Search/Accuracy’ to the right hand side (see image below) for maximum returns (if you leave it at the left hand side, you’ll be limited to 12 results).

We are now ready to obtain a list of @HidingIt‘s friends, who are the people Tony Hawk enlisted to hide #THTH packages around the world. We can achieve this by right clicking and selecting the ‘To friends of this person’ transform (see image below) as we did when we generated @SweetJerome ‘s friends earlier.

Here what we end up with after running the transform; 84 friends returned.  (note: we can ignore @SweetJerome and @tonyhawk for now)

#2. Try getting the tweets of one person, let’s start with me, @TheSuggmeister. (we’ll mine these later to figure out locations)

Select the AffiliationTwitter @TheSuggmeister, which has been converted here to his name, Suggy (see image below).

Right click on ‘Suggy’ and select the “To Tweets [Search Twitter]” transform (see image below).

Happy days!……………….Well not quite.

As of writing this, I had tweeted roughly 525 times (blimey!). Yet using Maltego only returned 55 results. ARGHH!… let’s be more accurate. the Twitter API returns 55 of my tweets to Maltego. See for yourself…

http://search.twitter.com/search?page=1&q=+from%3ATheSuggmeister&rpp=100

Note: that last number varies the number of results returned to that page.

So, what’s going on here?


In the blue corner – The Twitter API (both of them)

The Twitter API is in fact two API’s, the Twitter created REST API and the Search API, formerly ‘Summarize, Inc’.

- The Twitter REST API appears to work as advertised.

- The Search API appears to suffer from some pretty well documented issues. The following article captures the issues pretty nicely.

http://blog.louisgray.com/2009/05/twitter-search-engine-is-very-very.html


Update

After posting this article I almost immediately spotted this from @singe

“Frustrated with the short term memory of search.twitter? Use “site:twitter.com/<username> <search term>” on Google for a longer view.”

…and here’s a link to his 2007 article on Data Mining twitter: http://singe.za.net/blog/archives/976-Using-Maltego-to-Data-Mine-Twitter.html.


It get’s more interesting…

While you will get ~55 results from a search on @TheSuggmeister (maybe more, maybe less), you will currently get ZERO results when searching for tweets written by ‘tonyhawk’. This is a big deal to my tiny project as @TonyHawk is pretty important to it’s success. In summary, not everyone is indexed and those who are indexed, are only indexed up to a point. So, as far as the API’s go, we can conclude that the REST API might be useful and we can pretty much forget about the search API (at least at the time of writing).

In case you’re interested, here’s a list of the all twitter API methods and how to call them, REST API v Search API.

http://apiwiki.twitter.com/Twitter-API-Documentation

So let’s have a look at which Twitter API calls Maltego uses in it twitter transforms. This should help us determine what might be useful to us and what will be… erm less useful. REST API = Good; Search API = Not so good.


Maltego Transform

Twitter API used (I’m guessing)
To friends of this person REST API
To followers of this person REST API
To Tweets [Written to this person] Search APIe.g. http://search.twitter.com/search?page=1&q=+to%3ATheSuggmeister&rpp=100
To Tweets [That this person wrote] Search APIe.g. http://search.twitter.com/search?page=1&q=+from%3ATheSuggmeister&rpp=100
To Person [Convert] REST API
To AffTwitter [This person wrote Tweets to ?] Search API and internal Maltego magic
To AffTwitter [This recieved Tweets from ?] Search API and internal Maltego magic
To AffTwitter [Get details of ID holder] Search API and internal Maltego magic


Clearly the use of the search API presents a bit of a problem
, but is Maltego down for the count?
Nope!


Enter Local Transforms

Maltego is highly extensible, providing the functionality to write your own transforms, in pretty much any language you desire and then call it/them from Maltego to do whatever you need. As the Paterva website states regarding Maltego local transforms:

“transforms that are executed locally and not on the server. These allow for custom transforms to be written that integrate with Maltego though a simple interface with the ability to code in practically any language. Sample libraries and a few custom transforms are available on the forums (http://www.paterva.com/forum/ ) under the “Development” section.

We also have the specification for local transforms available on the wiki (http://ctas.paterva.com/view/Specification ) as well as in PDF version (http://www.paterva.com/docs/localTransforms-SpecIII.pdf ).”


Source: http://www.paterva.com/web4/index.php/news/128-maltego-202-release-with-local-transforms


So what?

So I could write my own transform, in my favorite scripting language (guilty pleasure = PERL), to interface with the Twitter REST API. FWIW, Getting data from the REST API in a cinch.

Try this to get my latest 200 tweets (note: results are limited to 200 per request).
http://twitter.com/statuses/user_timeline.xml?id=thesuggmeister&count=200

…and the acid test. Does this work for ‘Tony Hawk’?

http://twitter.com/statuses/user_timeline.xml?id=tonyhawk&count=200

You betcha!

There’s mountains more information about calling these API’s at the following links (I found these most useful anyway. I’m sure you have your own favourites).


Buyer Beware

Here’s four things you’ll have to consider when using the REST API as a crude search too.

  • The 200 tweet limit we already talked about.
    • We can probably ignore the 200 tweet limit if we search between some dates as we can assume that no one will tweet 200 times in, say, one week.
  • You can’t search by date (directly).
    • Whilst we can’t search by date, twitter appears (as far as I can tell) to index tweets with an ever incrementing ID. Outer limits are defined with since_id and max_id (see here). Here’s some approaches I could also have taken:

      • I could have written some PERL to recursively work back from a known date/time, in 200 tweet chunks until I’d reached the 1000 node Maltego limit or ran out of tweets;
      • I could also have written a small tool to go through someone’s timeline to get a rough idea of twitter ID’s on a given day;
      • I could have written something to timestamp tweet ID’s. I couldn’t find a “Convert Date/Time to Tweet ID” out there, but it wouldn’t be difficult to put something crude together. In the end I used the “finger in the air and guess” approach, which was good enough for my little project. i.e. I made an assumption that no one would tweet more than 200 times in 2 weeks and therefore went back in roughly 2 week chunks (finger in the air approach).
  • The REST API is limited to the last 3200 tweets.
    • For this project I could ignore the 3200 tweet limit. However, if I return to this project in a few months or years, this might then become a limitation.
  • By default, you are limited to 150 calls per IP address, per hour.
    • The 150 calls per IP address are a problem you’ll need to work around, as my local transforms make one call for every twitter user identified. So, if I ran the local transform on @tonyhawk, and he referenced 149 people in all his tweets going back to Oct 11th 2009 (ish), we’d be safe. However, if he referenced 151, we’d have a problem to work around. This also means that if we run the local transform on multiple entities (e.g. @tonyhawk AND @TheSuggmeister) their combined twitter user references would need to come in at under 150. This quickly becomes a problem.
    • After some pondering, I requested that my IP address be whitelisted, which raises the calls per hour from 150 to 20,000. That’s right folks 20K API calls, every hour!  Again, another piece of amazing luck! “Your request for Twitter API whitelisting has been approved”.  You can read more and make your own requests here:
      http://apiwiki.twitter.com/Rate-limiting
      (Note: I’m not sure if you can just request a whitelisting for your twitter user account since I had a static IP anyway).
    • For my tiny project, my code made nearly 10,000 calls to twitter in order to map out communications for just thehiders. That’s a non trivial amount in my book.

Writing the local transform

Rather than explain my local transform, here’s a link to the commented PERL script I used (Caveat: the code isn’t pretty in the slightest).


Putting it all together

1. Find Hiders
Derive @HidingIt (as we did  earlier)


Get friends of @HidingIt using the built in ‘To friends of this person’ Transform

2. Find the Finders
Get Tony Hawk (he’s one of @HidingIt’s friends).


Run my ‘GetTweeteex‘ local transform to get all the people @tonyhawk referenced in tweets going back to October 11th ish 2009 (when his twitter hunt started).

Here what we end up with after running one of my ‘GetTweeteex‘ local transforms (u see 4 pointers, here, when we run all 3 xforms you’ll see)

…and this is what our map looks like after running all three ‘GetTweeteex‘ local transforms on @tonyhawk (see image below). At this point we have a graphical representation of all the people @tonyhawk references in his tweets going back to roughly October 11th 2009. Here’s a Maltego Map of this stage, so you can play around.

Do you notice something? The entities (tweeple) with red lines pointing to them are referenced in tweets sent by @tonyhawk and are also friends of @HidingIt. This is where the real beauty of Maltego starts to emerge; i.e.  we can already infer that something interesting might be going on (note – Eddie  Elguera is a pro skater, Suggy – aka TheSuggmeister is yours truly, Jerome Case is one of Tony’s employees/side-kicks and Shana Hartman got engaged shortly after the THTH event to earn her tweet – congrats BTW).


3. Run ‘GetTweeteex’ local transform to see who spoke to who

This is already looking promising, so now we want to proceed and figure out who spoke to who.  For each friend of @HidingIt, we need to get all the people they referenced in tweets going back to Oct 11th by running the ‘GetTweetee’ local transforms.  Warning, this is where you’ll really hit your twitter rate limit unless you’re introducing a delay in your transform, some other workaround (make sure your Maltego timeout is changed accordingly), or get whitelisted (given the number of calls you’ll make, I’d recommend the latter).


The image below shows the results after running the ‘GetTweetee’ local transform on just a few
@HidingIt friends; some relationships are already surfacing and Maltego lets those of us who prefer to look at data visually quickly see what’s going on.

Continuing to run all 3 ‘GetTweetee’ local transforms on the friends of @HidingIt leaves us with this (see image below). Here’s the Maltego Map of this bit too.

One thing you’ll notice is that there’s A LOT of information here and even for visualization geeks, this is a bit of a stretch.   We should, however,  see some links between hiders and seekers (I switched to the centrality view to see show the links, more about that shortly).


Note: I realize that you can’t see some of the names in the diagram/screen shot below; That’s not too important here, the key this is that this bigger the circle (entity), the more stuff is going on.  We can already see (no surprise) that @tonyhawk andr @HidingIt are the big ticket items, but there are some other interesting things beginning to emerge.


4.Prune the tree.

In order to make sense of the graph above we really need to cut out the relationships that’s were NOTinterested in.  In this case I only care about communication from the people @tonyhawk references (the potential finders) to/from the friends of @HidingIt (the hiders).  I therefore want to cut out all the extraneous chatter (the noise, if you prefer Signal/Noise metaphors). i.e. Cut out all of the people further than one link away from @tonyhawk and/or @HidingIt


A little bit of effort later and voila!  a pruned tree.  Great, what did I see?


Different Views & Layouts

Maltego provides a few different ways to view your data. The Maltego wiki has a full section on views and layouts, so there’s no need for me to repeat it all. Here’s the link. http://ctas.paterva.com/view/Userguide#Views_and_layout

“Views are used to extract non-obvious information from large graphs – where the analyst cannot see clear relationships by manual inspection of data.”

Source: Maltego Wiki

Mining view

This is pretty much the default view and the view you’ll use to interact with your map (running transforms etc… i.e. mining).


From here I can switch the layout (see wiki for info on layouts), making it easier to view relationships (this layout is the Centrality layout) and explore more and more.  Note, we’re still in mining view, so we can still run transforms. Don’t confuse layouts with views.

This mining view shows the Organic layout.

Centrality View
“Centrality view. Nodes that are calculated to be most central to the graph are given larger nodes”

From here, I get a nice overview of who’s talking to whom and can clearly see who’s central to the graph.  In our case it should be no surprise to see  @tonyhawk and @HidingIt

Edge Weighted View
“Edge weighted view. Node sizes are based on number of incoming links”

As the Maltego userguide points out “we see something more interesting” , all of a sudden we can CLEARLY see some entities (tweeple/people/whatever) who might be worth having a closer look at.


From here I can see who chatting/being chatted to the most and dive down into the detail.  As I’ve stated previously, this is the real beauty of the product.  Sure, you could do the same thing with spreadsheets and sort by number of incoming/outgoing tweets,  but (if like me) you’re more Visual than Auditory, Kinesthetic, Olfactory or Gustatory (Neuro-linguistic Programming for Dummies, Chapter 6 – if you like this find of stuff, check out Richard Thieme & his DefCon talks, they’re EXCELLENT), you’ll find this way much more agreeable.

Here’s the same image as above, but with some red circles to highlight the areas where interesting stuff may or may not be going on.


So what does this actually tell me. Well, it tells me that even if I hadn’t been following the #THTH event, I can see that the following were pretty active talkers/talkee’s about #THTH in comparison to others

If you (with no knowledge of the event) had determined this, you’d probably have drilled down to my blog and read the articles that myself and @Steven_Gill wrote. You’d have also been able to follow the links on my blog and read other hider/finder stories.  You’d also have figured out that @SweetJerome pretty much ran the event for @tonyhawk.  Just with these 4 pieces of information, you’d probably know everything about the #THTH event you could ever possibly want to know.


Did it help me get the Google map?

Hmmm, to a very limited extent (erm, almost not at all actually). But,

  • Generating Google maps is not really the point of Maltego. The point (for me at least) is to quickly identify areas of potential interest. i.e. The tool quickly showed that it would be worth finding out a little more myself and @Steven_Gill.

My efforts were also thwarted by some of the following:

  • Not all hiders/seekers had public timelines, so it was difficult to see where they were located (I didn’t delve into authenticating my session).
  • It looked like some of those people had deleted tweets, so I couldn’t find their messages to Tony about where they were located and why they should be trusted.
  • People didn’t always use the #THTH hash tag (did they Mr. Tony Hawk?… I’m lookin’ at you mister! ;-)    )
  • Finders didn’t all tweet to @IFoundOne, which really complicated matters.
  • It appeared that Tony had enlisted some people to help hide things who weren’t on twitter.
  • Some packages went missing in action.

So, @MrsSuggmeister and I finished off the job whilst in hospital waiting rooms. We tackled the gaps using two spreadsheet print outs and a pen and paper.

The final step was to create the Google map. I could have scripted it, but in the end, with a little (a lot of) help from Mrs S, we just used cut & paste.


Lessons Learned

  • Have a plan. Sketch out what you want to achieve with Maltego. See these pics for an idea.



  • Pic #1. General Concept Pic #2. @tonyhawk Pic #3:
    @tonyhawktweets announcing finds



    Pic #4. @HidingIt and friends Pic #5. @tonyhawk and people he tweeted Pic #6. Relationships between people @tonyhawk mentioned in his tweets and friends of @HidingIt

  • Try it with the search bar set to the left, that way it’ll only return a few hits (~12). Which is very important when talking to twitter as the API is rate limited.

  • Once you’re happy, let it roll on full tilt mode (Note: it’s not called full tilt mode, but maybe we Paterva could
    consider that term?)
    :-) .

  • If you ever think, “that’s weird. Not many results”, it’s probably because your Speed/Accuracy bar isn’t over to the right.
  • Local transforms open up a world of opportunity with this tool.
  • If you’re going to leverage Twitter API heavily, you should really consider making a whitelisting request.
  • Data mining twitter is a lot easier than I would have guessed 6 weeks ago.
  • The Paterva people are thoroughly smashing people and appear very committed to constantly improving theirsoftware. They had no obligation to help me, but they did.
  • If you really want to harness the power of the product you might seriously considering the server platform. Why? Hosting your own server means the transforms are running from that, and not a server hosted by Paterva and thus, if a transform uses an API, you will not be competing with anyone else in terms of rate/call limitations.


In closing

You can use this tool straight out of the box and get some very interesting results data mining all sorts of things. See these excellent articles by

Spend a little time digging deeper and I think this tool becomes a key addition to a pen testers tool kit. While I mention pen-testers, this tool certainly isn’t limited to use by testers. With the local transforms, it’s uses are both far and wide.

I can’t wait to see what’s in their next release and the 2.1 Alpha already looks amazing.


Special Thanks/Credit

Thanks to Roloef Temmingh and Andrew MacPherson for your excellent collaboration on helping me accomplish this mission.

Thanks to Twitter for approving the whitelisting request. which was as unexpected as it was awesome.

Thanks to @alien8 (reviewing my article) and @l0sthighway (pointer/help with Maltego) from DC4420 (great bunch of geeks doing some cool stuff and eating pies in the process).

Thanks to Tony Hawk and Jerome Case for indulging my nerdy interests and switching me on to the potential of twitter. Thanks also for sending our little boy *the* perfect “Welcome to the World” gift. He’s going to love it.


The Suggmeisters with the deck Tony sent and “Dave”, the reason it took me forever to finish this article.


Charity

If you found this article useful and/or interesting, please consider donating a few dollars help Tony Hawk build a skatepark for the youth of Watts. The Tony Hawk Foundation will match donations until the donations reach $10,000.

http://www.globalgiving.org/projects/tony-hawk-watts/

“Tony Hawk has joined the youth of Watts to create a skatepark that will promote a healthy lifestyle and be an alternative to the crime, gangs, and drugs that pervade their challenged community.”

One community at a time… isn’t that right Tony  :-)


References

Maltego References:

Article and Hyperlink Author(s)

Maltego and Twitter

Paterva

Maltego 2.0.2 release (with local transforms)

Paterva

Maltego 2 and beyond – Part 1

Mubix

Maltego 2 and beyond – Part 2

Mubix

Maltego 2 and beyond – Part 3

Mubix

Maltego for

Information Gathering Part 1

Carnal0wnage

Maltego for Network Infrastructure Enumeration

Carnal0wnage

Maltego Malware Domain List Transformations

Carnal0wnage


Using Maltego to Data Mine Twitter

Singe


Information Gathering with Maltego

Agent0x0


Twitter References:

Article and Hyperlink Author(s)

Twitter’s Search Engine is Very, Very, Broken

Louis Gray

Using the Twitter Search API

Brian M. Carey
, Information Systems Consultant, Triangle Information Solutions

Twitter API Documentation

Twitter

Twitter Search API

Twitter

Things Every Developer Should Know

Twitter

Twitter REST API Method: statuses friends_timeline

Twitter

Twitter Rate Limiting

Twitter