Daily Operations Glitch

[Confuscius]

This made me giggle!

I was running out of operations allowance so submitted a support ticket earlier this afternoon, fully expecting to run out about 4.00pm. Anyways having just checked my account and seeing that my sites were still live I was puzzled until I saw that my account now shows:

Usage
Daily operations allowance:
400,000
Remaining operations:
4,294,965,360

That should keep me going for a couple of days or so!

Clearly some tehnical glitch but at least my sites did not go down!

Paul

[Adrian]

Hi Paul,

Will have a look at this in the morning for you. Seems like a bug in the system!

Thanks for letting me know,

Adrian
SW Integration & Technical Support

adrian.botelho@digitalwindow.com
www.digitalwindow.com

[Adrian]

Hi Paul,

This should be sorted for you now, please let me know if you have any more problems with this.

If you need your quota further reviewed, please drop me an email.

Cheers,

Adrian
SW Integration & Technical Support

adrian.botelho@digitalwindow.com
www.digitalwindow.com

[Confuscius]

Hi Adrian

Thanks for the update and operations increase - already looking like it won't be long before I blow the latest limit! Don't you just love the predictability of search engine spiders!

Paul

[sspyrou]

Hi Guys

Looks like im not the only one that keeps nailing my quota.. its getting to be a pain as i typically run out between 10am and 1 pm - just in time for the daily peak.

Analysis of logs has led me to conclude that this is in part due to rogue spiders / scrapers etc including spambots - mostly from foreign ips.

Lets set up a thread somewhere where we can share and ideas for conserving quotas..

Here are a few to start....

1 - Robot exclusion. Block useless, well known bots.. (feel free to refine the list to your prefences)

User-agent: Googlebot
Disallow:
User-agent: msnbot-media
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: MSNBot
Disallow:
User-agent: Teoma
Disallow:
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: *
Disallow:
Crawl-delay: 20
Disallow: /cgi-bin/

User-agent: ShopWiki
Disallow: /

User-agent: psbot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: *
Disallow: /

[sspyrou]

2 - I find that yahoo is a pain these days - it consumes bandwidth and returns little in the way of useful uk traffic that converts through shopwindow.

This may however be just for me....

I begaan by blocking it from the many product pages using robot.txt

User-Agent: Yahoo! Slurp
Disallow: /product.php?*
User-agent: Slurp
Disallow: /product.php?*

However, this has little effect on reducing my calls

[sspyrou]

3 ) All good spiders obey robots.txt

Using the craw delay command (look it up in goodle for more info) should slow down the rate at which these spiders consume your quota on newly launched sites...

Add..

User-agent: *
Disallow:
Crawl-delay: 20
Disallow: /cgi-bin/


to the robots.txt file

[sspyrou]

4 ) Here is a great way for blocking pesky scrapers and automated bots from running rampant through your site and consuming you quotas

#http://www.spider-trap.de/en_installation.html

Basically, set up a spider trap.. and then exclude them from yor site using htaccess...

[sspyrou]

5 ) An OK tool... (Ive not compared it against other similar services) for blocking visitors by is http://www.blockacountry.com/.

Be warned, its not 100 percent perfect... If anyone has a better ip list let me know.. else i will provide a better list in time if there is any interest in this thread...

Basically, the idea here is block out, or redirect visitors from contries that are unlikely to achieve anything other than attrition of your quota.

DO NOT BLOCK OUT THE USA! if you do, google, yahoo and nsn will not index your site!

Use at your own risk

[sspyrou]

I've started replacing some of my category treas with static menus in the hope that this reduces by calls by ropughtly 1/12, based on each page load resulting in about 12 SOAP calls...

I need a little clarification here.. Im good at manipulating bots, but not great with php / SOAP.

I have assumed that if i rework category_tree.tpl, the calls will cease. Is this the case or is this an incorrect assumption? Are the calls initiated by category_tree.tpl or a differnet script?

Thanks

S

[Bluesplayer]

Darn - happening to my sites now too.

This quota limit a permanent fixture?

[GeorgeGaz]

Hello,

Please see this thread (http://www.shopwindowforum.com/showthread.php?t=1219).

Cheers

[Andy]

re: this bit:
Our aim with the re-haul is not to switch affiliate accounts off or defer them to other solutions. Our aim is simply to ensure an optimum service can be delivered to all using the service and this has a large requirement on us being able to measure and control usage.So we're not all getting turned off.
Then this bit:
Essentially, we want you guys to generate more commissions from the use of the platform, and this is what our work is being based on. If you're in a position where you're not quite hitting the mark, we will have other ways that you can continue your work with us.
Oh, so we are all getting turned off then?

And if you're just cutting the quotas, What if there's a google algo update and all the big guys you have kept on (that much is obvious) suddenly get dropped out of results leaving just the little guys with 37 calls a day of quota?