May 11, 2010
Google shafts Scroogle
I attempted to do a search using my favourite search website today – Scroogle Scraper – only to be redirected to the following post on Scroogles homepage:
We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we’ve been scraping since Scroogle started five years ago. Actually, we’ve been using that interface for scraping since Google-Watch.org began in 2002.
This interface (here’s a sample from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.
That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn’t show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google’s basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched “Universal Search,” which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.
Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google’s main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.
Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.
Thank you for your support during these past five years. Check back in a week or so; if we don’t hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.
— Daniel Brandt, Public Information Research, scroogle AT lavabit.com
http://www.scroogle.org/cgi-bin/nbbw.cgi
- accessed 10:10am, Tuesday May 11th 2010
If, as suggested above, this really is the end of Scroogle, then this change marks a sad day.
What was important for me about the Scroogle Search, was that it offered an alternative (parasitic?) reading of Googles search algorithm, providing results free from paid advertising and also without granting Google an IP addressed search record as exchange currency.
I have to admit that I won’t miss the hokey cartoons and seemingly abject anti-Google paranoia which prevailed on Scroogle, but my internet experience will be poorer without their regular and healthy dose of skepticism toward Googles globally dominant infrastructure.
The example set by the Scroogle Scraper offered the potential for alternative and critical readings to flourish online, taking datasets and information provided by major institutions as ’source material’ rather than ‘finished product’. Without the option for ‘interfacing’ with the information, that potential is diminished: it wasn’t ‘the vandals took the handles‘ after all.
(Please add comments below.)
I think I disagree with the Google Scraping project fundamentally. It’s approach is flawed, and it’s ideology was never on stable ground.
For instance, you state that one of the reasons you liked it was freedom from a singular trackable IP address for Google – now, without looking into the technical side of things, I don’t know what’s going on. But it would be foolish to say that Google couldn’t log results from Scroogle. Scroogle would also have to have some form of IP tracking built into it’s web service somewhere, even if they never look at the numbers.
While this service existed, it lulled less-technical users into thinking that they were protecting their privacy. They weren’t. Where they investing in VPN networks, such as http://www.ipredator.se, or recommending the use of proxies to search with? No. And without doing things like that, your browsing on the internet is open to anybody sharing the same access point, or to your ISP (who have no choice but to legally snoop on you if certain types of traffic are detected).
When ‘easy’ services that promise a false sense of privacy – like this Scroogle service – stop working because they were reliant upon one technical trick, it shows that what they offer isn’t as comprehensive as claimed.
Hi Pete,
I completely agree that in technical terms, Scroogle was a pretty weak one-trick-pony, and that its offer was further hampered by being totally entrenched in the privacyprivacyPRIVACY! camp.
However, my own interest in Scroogle, and the reason that I have written this post, is that Scroogle offered, for me, an alternative ‘voice’ thru its re-interpretation and expression of Google Search Results. Individuals were not forced or mis-directed to Scroogle, but if they chose to use it they could to receive Google results, ad-free, thru the Scroogle Scraper filter.
Yes – the voice of Scroogle may have been cranky, oppositional, angry and at times annoying – perhaps even, as you say, poorly informed – but, yesterday, that alternative, critical, voice was snuffed out.
If I had to use a metaphor, I’d say that Scroogle were the UKIP of privacy groups. I might not agree with them, but I know there are people with far more advanced thinking on the subject.
By allowing an alternate interface to Google they themselves were guilty of not encouraging adequate security (VPNs etc). Indeed, they seem to have a somewhat vested interest in making sure that people only use their site to maintain their ‘privacy’. There’s no links to an alternate method of maintaining privacy now that their system is down, no recommendations of what to do instead of Scroogling…
Perhaps this isn’t so much an alternative as a competitor, or (as it uses Google’s own technology) a parasite? It’s not hosting another option using one of the Open Source options that are available to it, so I can only assume that they are not truly committed to an alternative.
Pete above doesn’t really seem to understand Scroogle. In his comments he seems to strongly imply that Scroogle provided “false security” or some sort of service which is easily worked-around by Google.
Scroogle acted as an anonymous proxy built to relay Google results. As an anonymous proxy, Google could only log IPs from Scroogle, which did free broadband individuals from being personally profiled by Google. At best, Google could only profile an aggregate of Scroogle’s user base.
It’s a small thing, but — with Scroogle — it was also an easily preventable privacy concern.
The false security that Scroogle gives is in the mind of the user – why bother to learn what a anonymous proxy is, when there is a web service that does it for you? I don’t think that’s a wise attitude to take.
Personally, I wonder how deep Google can profile. For instance, a collection of related search terms within a similar time period will probably be the work of one person. I have no idea how much Google can glean from these tiny pieces of data, and for me personally the tools that they offer are worth more than the inconvenience of being locked outside Google’s data wall. For what it’s worth, Google offer a service to move data over that wall too: http://www.dataliberation.org/
Anyway, Scroogle’s back up now:
http://www.theregister.co.uk/2010/05/12/scroogle_returns/
(From Scroogle) -
July 1, 2010: Here we go again…
We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we’ve been scraping since Scroogle started five years ago. Actually, we’ve been using that interface for scraping since Google-Watch.org began in 2002.
This interface (here’s a sample from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.
That interface was at http://www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn’t show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google’s basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched “Universal Search,” which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.
It is not possible to continue Scroogle unless we have a simple interface that is stable. Google’s main consumer-oriented interface that they want everyone to use is too complex, too bloated, and changes too frequently, to make our scraping operation possible.
After a lot of suggestions from Scroogle users, and a fair amount of publicity, we found a fix and Scroogle was back in 24 hours. This fix was to insert an extra parameter, &output=ie, into the search terms that were relayed to Google. The extra parameter recovered the same interface that we thought was gone forever.
Now it seems like it actually might be gone forever. Late on June 30, 2010, the results produced while using this parameter began to shift to the usual busy Google interface with ads and a left-margin sidebar. Scroogle users saw a Scroogle page that said, “Google returned no results for this search,” when in fact Google returned results but our scraper was unable to deal with them. Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.
Thank you for your support during these past five years. Check back in a week or so; if we don’t hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers.
— Daniel Brandt, Public Information Research, scroogle AT lavabit.com
I am one of those people that, try as I might, am lost when it comes to understanding what to do to protect my privacy online. I have read about proxy lists, (don’t really understand what you’re supposed to do with them) anonymous proxy software , tor, onion, etc. and still I’m lost. I’m a reasonably intelligent individual, but being 53 years, I did not grow up with computers, the internet, etc. and it’s like my brain was never hard-wired to understand these things.
I have tried and still not found one single comprehensive source to explain these things to me. Whenever I do read something, the terminology used is so unfamiliar it’s almost akin to reading a foreign language and being expected to understand not only it, but the very technical information it is attempting to impart.
For those reasons, I was happy to find Scroogle. I knew it was not an end-all-be-all to internet privacy, but it really cheeses me off that groups like google spy on us. I don’t have anything to hide, I lead a pretty boring life, actually. But when I get in to my car and drive to work, go for lunch at the corner cafe, pick up my dry cleaning, stop by the bank, pay a bill at the electric company, etc. I am not being followed and my every move is not being watched and recorded. If only the same were true of my online life. I don’t want google or anybody else tracking my every move so they can “give me a better online experience,” or showcase the ads they think I’m most likely to fall for. No thanks. Just give me my privacy and leave me in peace.
*Phew* Thanks for letting an old dog get a few things off his chest. Now, if any of you techno-wizard types could point me in the right direction, I’d love some advice on learning this stuff from the ground up.
In the absence of scroogle I am using Duck Duck Go here: https://duckduckgo.com/ They also do not keep browsing logs or search histories and they have an encrypted search function. So far it seems pretty good and uncluttered.