Rants & Essays

The Unsearchable IDT

By Marcus Pan

IDT's done it again…added yet another policy to its collection. In the process, they have rendered their entire server (and all sites held within including the Inferno Publishing site (http://www.idt.net/~pan)) unsearchable to most of the search engines out there. How did they do it? It's rather simple, really. In order to understand it better, we'll have to discuss how search engines work a bit, though.

Search Engine Robotics
Search engines on the World Wide Web (or most of them, anyway) have ways that you can submit your site's URL for inclusion in their databases or indexes. Usually, it only takes the entering of the URL itself and that's it. After receiving this, the engines send out "robots" to grab the pages of the site that was entered. These robots simply surf over to that site, pull down the HTML, and then index it so that it is now searchable by key words. Robots are simple little things…they don't care much for what's there, just the fact that they can get to the page entered. Some are smart enough to delve further into the site, following the links so to speak, so that eventually the entire site and all the pages within are in the database.

Access Denial
IDT's newest policy involves the use of a ROBOT.TXT file on the IDT server on which users have their web sites. This file restricts access to any and all robots from all the various search engines. And when a search engine robot comes through to a site and is not allowed access, it is unable to pull pages for its index. And when it can't index a page, it can't submit it into its database. Basically, that boils down to one thing. Any sites held on IDT's server, even those that are supposed to be business sites, are unable to be added to most of the search engines out there. So much for being pulled up by millions of Internet users looking for information that you may be able to provide them…

No Notification
Of course IDT hasn't told any of us about this. And it is a newly added "feature" to their slew of shitty Internet services. The policy was put into effect, and the ROBOT.TXT file added, some time during June or July of 1997. I know this because I was able to register and index any sites I have on IDT up until this time. I had no problems before. IDT hasn't notified their users as usual, so this will probably come as quite a shock to their them…surprise, surprise. But their constant policy changes has become the staple on which IDT prides itself. They seem to think they can change their entire service around without telling people who use this service any time they have a hissy fit.

The Ender of Support
Let's add to this some recent experiences I've had in IDT's #IDTSUPPORT channel on IRC. I logged in to get information about this newly added "feature" of their service. After spending well over a half hour discussing this with some EnderK fella who calls himself a technician, including discussions about how I have a business account and not being able to index, add and therefore advertise my business site in most of the search engines Internet users go to for information, around midnight he stated, "Shift change. Bye." What the hell? So he left. I had to re-explain my entire position to another, similarly challenged and unhelpful technician. And, by the way, Beefy Brad still won't talk to me-the wimpy little so-called "administrator" that he claims to be.