Future Work and Improvements

Build a Twitter Analytics App

1 Introduction: Start Here

2 The First Step: Design Your Solution

3 In Which I Rant a Little

4  Design Solution

5 Writing Great Code

6 Writing the Backend Twitter Server

Writing the Code in Small Parts: Part 1, The Basic App

Part 2: Adding a Counter to Exit

Part 3: Adding Language and Retweet Count

Part 4: Organising Our Code

7 Adding the Data to a Database

8 Testing: What and How to Test

8.1 Testing Our Frontend

8.2 Testing Our Backend

9 Displaying our Data using the Flask Webserver

9.1 Introduction to Flask

9.2 Adding templates to our Flask app

9.3 Displaying our Tweets in the Flask Web Server

10 Future Work and Improvements

Okay, so this is the last section. You should have a bare bones Twitter Analytics engine working now.

But if you compare it with mine, http://www.twitteranalytics.org.uk/, you will see a few differences.

One, mine is just prettier 🙂

The reason is simple: I just used a Bootstrap theme, specifically this one. You just take your existing HTML code, and copy paste it in the Bootstrap theme files. There might be  a bit of moving/cleaning stuff around, till it looks the way you want it to look.

You will also note that I do a lot of other stuff. Like, checking for countries, swear words etc. The thing is, once the logic is there, adding all this stuff is easy. I just search for these terms in the Tweets.

The code for the live website is actually here. It’s not as clean as I wanted it to be, but I wrote it in a hurry, so that I could demo the concept. Have a look.

The code still will have bugs. Remember our try-except block in the Twitter streaming class, where we were doing a pass because we got some malformed tweets?

Well, now that we have a statistics class, we can log those error tweets to a file. That way, we can analyse them and study how to work around them without skipping them. Of course, not all can be parsed. Many of these bad tweets are deleted tweets (which is strange  to me- why are deleted tweets being displayed in the first place?), so we can do a check like:

I won’t do that for you, but you have enough info to do it now.

You can make a lot of other improvements / add features. Like:

1 Check if the Tweeters are men or women. Most people dont share this info, so you’ll have to use other methods to guess, like their names.

2 Check where most Tweets are being posted from. Again, not everyone shares his info. Could you guess from their tweets?

3 Try to guess the age of the people tweeting. As a simple rule, anyone tweeting about Justin Bieber is less than 18 🙂

Let me know what progress you make, or if you get stuck.


PS: Want a free 1+ hour video course, Introduction to Web Scraping and Data Analysis? Also get a free mini-book, Python: From Apprentice to Master.

* indicates required

I will never spam you. Unsubscribe anytime.