Build a Twitter Analytics App
6 Writing the Backend Twitter Server
9 Displaying our Data using the Flask Webserver
If you were to ask a new programmer what the first thing you do is when you start a project, they will usually say Start banging on my keyboard! Like a monkey is implied.
Others might say, Aghhh, no. You write tests first.
Why you must Design First
Design has gotten a lot of bad publicity. Mainly from companies that used the waterfall process. Many companies that claim to use Agile are still influenced by waterfall. In waterfall, you did each step one after the other. So, design, coding, testing, release. Once you were done with design, you never went back. So you made sure you did it in excruciating detail. Hence 500 page design docs that had 50 pages of This page has been left intentionally blank.
When Agile came along, many people said, Woohoo! We are done with design. No more documenting.
Which is a mistake. In Agile, you still follow the whole Design-Code-Test cycle, just you do it in 2-4 week sprints, not 2-4 year cycles. Small sprints means you can come back and fix any problems, with redesign if necessary.
Design doesn’t have to be big thick documents. In fact, no one reads them. As I will repeat again later, the majority, more than 50-60% of your document, must be diagrams. Thick blobs of text put your readers to sleep. Short and sweet diagrams will help put your point across, and make it more likely people will read your document. The text must be just clear up stuff that can’t be extracted from the figure.
So why should you always design first?
1. The Thinking Problem
The hardest part of any creative task is thinking. Thinking slows you down, makes you doubt yourself, leads to stalling and getting stuck in the What do I do next? loop.
Design fixes this, because the thinking part has been extracted to an earlier stage. You make all the major decisions here, so that when you come to coding, all you are doing is following a checklist.
2. No surprises
The important thing about a good design is that it should remove most surprises (not all, unless you are a psychic or Harry Potter) .
So you want to use MySql, but the server only supports a proprietary database. When would you rather find out- at design stage, or after you have 2 million lines of code written?
In the design stage, you try to anticipate any known problem, and try to find fixes for them. So that when you actually start coding, you know 70-80% of what you will expect to happen.
You will still face problems, still struggle, but not because you missed something obvious. Programming is hard enough as it is, without also chopping your own foot off by not thinking through the problem.
3. Hardware / software limitations
You will always have limitations. They may be technical, or human (covered in next point). Maybe you don’t have enough servers, or your servers can’t run the version of software you want. You need to be aware of you limitations. I had a big one that stopped me from using the libraries I wanted, as you’ll see in my design.
4. Human limitations
How many programmers do you have? What are their areas of expertise?
Since this is a learning project, chances are, it’s just you. What are you strong at? What are your weaknesses?
You must design this app. Don’t look at the link I shared– it is the final version, borne from many iterative cycles.
Start with this. A client / boss gives you this instruction: “I want a web app that will show me the top tweets on Twitter, plus which languages are most common. I’d like to see this in graphs, if possible.”
Think this isn’t detailed enough? You’d be lucky to get even this much detail out of many clients.
With this in mind, I want you to design your solution. Don’t go too crazy, just use pen and paper. In your design, you must consider:
- What blocks of code you’ll have
- What they will talk to (each other? A database?)
- How data will flow through your app
You must have enough detail so that you can give the document to someone, and they’ll know exactly what to do. Don’t go crazy though. Spend no more than 5-10 minutes.
If you have never done a design before, don’t worry. Try it anyway. I’m throwing you into the deep end for a reason. You learn better when you struggle a little before looking at other people’s solution.
Some of you are thinking, Meh. I won’t bother. How will he know if I’ve done it? If you are thinking like this, you are wasting your time. Just looking at my solution won’t teach you anything. And if you aren’t here to learn, why are you here?
So spend 5-10 minutes now, using a pencil and paper, drawing a rough design of how you think the solution should look like.
There is no right and wrong here, remember our design will evolve anyway, as we know more about our project.
Next: Look at my design.
PS: Want a free 1+ hour video course, Introduction to Web Scraping and Data Analysis? Also get a free mini-book, Python: From Apprentice to Master.
I will never spam you. Unsubscribe anytime.