I agree 100% with what you have written here. It is a hard choice to make as far as defriending people but sometimes it has to be done.
I used to follow people that have hundreds of followers on Twitter and only followed back less than 100. I finally stopped following them when I noticed that the number of people they were following had not changed in the last 4 or 5 months. To me, this meant that they had no intention of following any new people so I stopped wasting my time on them.
Now I have more time to develop new relationships with people whom are still interested in getting new followers and deepen my relationship with people that I currently follow.
On that note, thanks for your friendship and for the follow on Twitter.