Part 2: Adding a Counter to Exit

Build a Twitter Analytics App

1 Introduction: Start Here

2 The First Step: Design Your Solution

3 In Which I Rant a Little

4  Design Solution

5 Writing Great Code

6 Writing the Backend Twitter Server

Writing the Code in Small Parts: Part 1, The Basic App

Part 2: Adding a Counter to Exit

Part 3: Adding Language and Retweet Count

Part 4: Organising Our Code

7 Adding the Data to a Database

8 Testing: What and How to Test

8.1 Testing Our Frontend

8.2 Testing Our Backend

9 Displaying our Data using the Flask Webserver

9.1 Introduction to Flask

9.2 Adding templates to our Flask app

9.3 Displaying our Tweets in the Flask Web Server

10 Future Work and Improvements

Again, we are developing our app in parts.

Last time, we were facing the problem that only a few tweets got printed.

This took me a fair time to debug. I suggest you spend at least five minutes on it, before looking at my solution.

The problem was that the live Twitter stream in wild and unpredictable. Normally, when you search for Tweets, you always get the same json object (and I hope you took my advice and spent some time playing with the json, so that you know what to expect). You get deleted tweets (why?!), tweets in strange formats. 80-90% of the time, the tweets are as you expect. We need to cope with the remaining corner cases. Here is our first attempt to fix it:

I’m only attaching the changed code. This time, we have the code in a try-except block. If the code finds a json it doesn’t like, it simply ignores it and moves on.

Before I do the code review, can you find anything wrong with this approach?

Code Review

This is a very dangerous anti-pattern. We are brushing problems under the carpet. I addition to bad tweets, any bugs in our code will also get hidden away.

But we do have a problem, that the streaming feed is unpredictable. We don’t our program to stop just because it found 1 bad tweet.

So what’s the solution? The best way is to log the bad tweets, and later on, see if we can write code to work around them.

However, we don’t have any logging facilities at the moment, so we will put this problem on hold, and come back to it later.

See the code here.

Adding a counter

Another problem with our code is, it keeps running non-stop till we kill it. We need to add some sort of a counter to make it stop.

You can’t add this to the on_data() function, as it is called fresh each time.

But we can add it to the class during initialization, and so it will be available each time on_data() is called. Let’s do that:


The init function is called when you create the class. We are passing in a variable num_tweets_to_grab, which is the number of tweets we’ll grab. There is also an internal counter. How do you initialise the counter? In your main code, you do:

We pass in a value of 10 num_tweets_to_grab when creating our class twitter_listener. Now, in the actual class, we do:

We return False, which causes the class to exit (this is how Tweepy works internally. If you return True, it keeps looking for new tweets). Let’s look at the whole code now (leaving out the imports). We have also included our code to search for tweets and trends here:

The code for this section is here. Make sure you understand what’s happening before moving on.

Next Part:  We will now start looking at the languages of our tweets, plus the top tweets.

PS: Want a free 1+ hour video course, Introduction to Web Scraping and Data Analysis? Also get a free mini-book, Python: From Apprentice to Master.

* indicates required

I will never spam you. Unsubscribe anytime.