View Full Version : Improving and Standardising Datafeeds - Discussion
To keep the thread George created for reporting feeds in order (click here (http://www.shopwindowforum.com/showthread.php?t=1052)) and so we can discuss the situation regarding products and bad category placement and all that gubbins, please use this thread so that we keep the other one clean, as has been requested by George.
My main gripe with this whole reporting feeds thing, is that certain affiliates on here have been doing it for as long as this place has been around, yet the exact same issues keep on happening, over an over, zero prices, wrong category, blank image, wrong price, html code, no line breaks, etc, etc...
It's all fine us affiliates doing lots of donkey work on this, but when the merchants don't bother doing their part, because they grown so used to expecting the affiliates to do it or for AWs to put some automated system that minimises what the merchant ever needs to do, it's no wonder there are so many fubars in the datafeeds.
Anyway, lets be having the rants and raves about this here. I'm sure they would be just as useful as the actual reporting of issues.
Thanks for this Amoochi :D great idea.
To clarify, within the series of scripts that we are running on descriptions HTML is one of the things that we are looking into.
Cheers
Confuscius
18-11-08, 13:31
<information>
Try this - http://www.shopwindow.com/productlist.php?rb=4-142|3-1219&iListOffset=10 - I posted here because I have no intention of following the required format because of the time involved and the distinct lack of commitment from Merchants in general to care one iota about any so called 'quality' or 'legal' issues.
</information>
<sarcasm>
How many of those products would fit a baby?
</sarcasm>
<hometruth>
This is the first category that you hit on going down the tree and the first sizable merchant that I picked and to be honest this is typical of the existing level of 'quality' control.
</hometruth>
<rant>
To resolve these issues using the methodology described would consume thousands of hours of unpaid affiliate time and I see no reason why we (affiliates) should be expected to fix the inadequacies of organisations who 'pay' people to sort out these sort of issues. The solution should be for Merchants to dismiss incompetent staff and employ competent staff.
</rant>
<action>
Off to make a cuppa!
</action>
Paul
Hi Paul,
What do we require from you?
You, the affiliates, know ShopWindow inside and out and what we would like is throughout your implementation, development and testing to flag to us any categorisation errors where you find products in the wrong category that should be elsewhere.
A good example of this can be seen here and the format that we would appreciate the error to be reported in is as follows:
I have emboldened some important aspects of the above passage.
If you prefer to use your own format, then please do and as long as it has the relevant information then I am sure we will be able to action it. I simply based that format from the format that we are using here as we do our investigating.
If you would prefer to not be involved in this at all then that is fine, I understand that you have other priorities especially in the run up to Xmas.
The solution should be for Merchants to dismiss incompetent staff and employ competent staff.
This may very well happen when we relay our findings back to merchants :D
Cheers
I don't mind helping out to improve the datafeed but why not give us an incentive by addressing a key issue that has been raised on this forum again and again in dozens of threads:
Hardcore porn DVDs appearing in the Entertainment category when the Adult Filter is on.
This has been going on since the beginning of the SW service and if I could do anything about it myself I would, but I can't. Please, please, please get someone to spend a few days flagging adult DVDs as adult.
I can't believe that an issue that probably causes offense to 99.9% of SW users, not to mention automatically contravening most programmes' T&Cs, doesn't warrant some kind of emergency manual intervention.
Hi authcode,
We are actually looking into this on a product level ;)
In the mean time we hope that identifying incorrect categories will help this procedure along.
Regards
Hi George,
I don't really understand what's to look into. The merchants responsible for adding these products (mostly Play, Zavvi and Choices) must know that these products are adult so why not force them to change the way they build their feeds to flag these items as adult in SW? I don't understand why that can't be done within hours given the severity of the problem. If the feed can't be built this way in a reasonable timeframe then why not just prevent these items from being included in the first place?
Obviously the category mappings are automatic in some cases and mistakes will happen but I don't consider a bread knife in the Finance category on par with hardcore porn in the DVD category. This is a special case that needs special attention, surely?
Hi authcode,
I completely sympathise with this situation and can assure you that this is being looked into with sincerity. The problem occurs with all of the correct procedures that we have to adhere to when notifying merchants, making changes, etc
This is where the time is incurred.
Now that we know what we want to do we are working on a solution but even this solution will only be as good as the data we are being passed. Getting development time has been tricky but we are making progress with this :)
Regards
I second that!
In fact, the finance section of Shop Window seems such a mess, I wish I could just remove it & only have consumer products shown on my site! :(
Hmmm Choices and Zavvi, two of the three biggest product placements needed for my main site, yet neither seem to be capable of including descriptions, making it even harder to tell whether their products are porn or not in the first place, there's a heck of a lot of subtly named adult products out there and i'm no porn pro...
So what's the case when there are no descriptions at all? Is this something being looked into? I thought the whole description thing was supposed to have been sorted long ago?
Merchant ID: 1815
Merchant Name: Choices
Link to example: http://www.amoochi.co.uk/productlist.php?rb=3-1815
Merchant ID: 1661
Merchant Name: Zavvi
Link to example: http://www.amoochi.co.uk/productlist.php?rb=3-1661
It's very difficult to determine whether products for either of those merchants are in the correct category or not, but I certainly aint gonna waste my time helping eother merchant improve when they've not been arsed to bother either making sure their date feed is working properly, or that they included descriptions for thier products in the first place. I've not even got round to looking at the other 600+ merchants I have dealings with yet.
Back to the actual subject at hand, just having spent a few mins looking at various products in the wrong categories for various merchants, I actually stopped counting at around 100 or so, when I realised, there's no chance in hell I can afford to spend the amount of time i'd need to, to actually list any reasonable quantity of these and that was with me just touching the surface.
I think AWs needs to get the tea boy off tea making duty and sit him down at a computer for a month going through the list of products all day every day and rope in a couple of dozen others to help him full time with it, because the kind of work needed to go through the product list and check the products is an impossible task.... Actually no, scrub that...
Having had my general look... I think this system of us writing merchants and all the details of badly categorized products down, is not going to work in a million years. It takes far too much effort first off and far too few affiliates are dedicated enough to take part in this when it takes such effort, never mind make the kind of effort needed to make any kind of decent impact. Also, as soon as there is any kind of sizable list of badly placed products posted to the thread (which believe me there would be), people would start posting duplicates and that is a complete waste of everyone concerned's time, as very few people would start going through pages of what is essentially database listings looking to see if their newly found badly placed product had been listed already. Personally, I think this needs a complete re-think and some kind of database driven type solution, where us affiliates can type in a product id and automatically have that products details pop up on the screen so we know we've selected the correct product, then we can set a flag like a tick box saying "wrong category" or something along those lines, that then gets added to a list that someone at AWs checks, which would highlight the flagged products to see if they are indeed in the wrong category or not. I'm sure AWs must have plenty of people there with the skill level required to do some work on the datafeed to achieve such a thing. Obviously it's something which would need moderating on an almost full time basis, but would be a damn sight more achievable and efficient than some kind of manual system like posting to a forum in the way we are being asked here.
I once worked for the UKs largest mortgage provider and was in charge of dealing with over 3,000 firms of solicitors and the literally millions of sets of title deeds (before it all went electronic and no paper) that each solicitor had in their office. Part of my job was to contact each solicitor to get them to return all of the title deeds that they no longer needed, that still belonged to the mortgage provider and all of this needed doing by hand. I had a team of 15 people under me and it was a full time project for them, sitting there just checking through title deed numbers, over and over and over. After 6 months of solidly doing this, we had got roughly 30% of the title deeds back. So... To cut a long story short, I know how much of a pain in the ass it is to manually do so much of this type of thing and I know full well that even being paid a good wage, people very quickly lose interest in it. So anyway that's just me supporting and backing up my "it's never going to work" theory.
Confuscius
18-11-08, 19:52
It may be easier to deal with this issue on a merchant by merchant basis based on severity of problem rather that randomly picking out individual products.
I for one would quite happily ban a complete merchant feed until such time as they resolved their issues. I would quite happily concentrate on one merchant at a time for review purposes and I am certain that if each affiliate interested in doing so was to be allocated a 'Merchant for Review' then we would get a clearer picture of how widespread mis-categorisation is.
On the other hand, if I know that a Merchant has a good feed properly categorised then I am more likely to offer advertising space on my non SW domains. A simple example would be the Play feed - in view of issues highlighted previously by others then I took the decision to 'ban' the merchant even though I had never personally examined their feed.
A collective approach needs to be sorted out instead of this shambles of a hit and miss approach - why were the categories that are going to be focused on chosen? My guess is to improve high selling areas from 'membership style' sites (a specific improvement in selected areas) rather than to provide what is important to 'seo style' sites(i.e. a general improvement across the board), but I am cynical!
Paul
I have to agree with Confuscius that this product-centric approach probably won't have a significant impact on the quality of the feed, given the vast size of the dataset.
I don't really know how the individual merchants' feeds are generated but from what I've learnt so far some kind of automatic mapping process is involved. I assume this mapping is on a category level and not a product level. Therefore the problem of mis-categorised products seems to be down to the mapping of merchant categories to SW categories. Right? So then why is this process automatic if it's causing problems?
Even merchants with millions of products will only have 100s of categories at worst, so why not manually map their categories to the most relevant SW category? This is the most logical approach and problems would only occur for products that were categorised incorrectly on the merchant end anyway.
I can't understand how luggage, for example, would end up in the travel insurance category? Presumably this luggage was in a "travel" category on the merchant site and an automatic process mapped it to the closest SW category based on a word match? If this was a manual, one-off process per merchant even an imbicile would realise that a more appropriate category would be "bags".
Anyway, I guess I'm grossly simplifying the situation. Perhaps if someone could explain the processes behind the SW feed some of us here could suggest some improvements? I don't think product-by-product correction is going to get the job done, unless you are using the data to tweak an automatic keyword-based mapping system??? Enlighten us! :)
Hello all,
We understand that this is not a perfect approach, far from it, but Rome was not built in a day. We are trying to take steps to improve all of this for everyone involved. Granted, the merchants may scoff at what we suggest but we are informing them of how this is going to help their programs.
Whilst you guys can ban merchants from your installs (and this review will flag some for you if we are allowed to name and shame a list) we do not have the authority to just instantly ban merchants as we need to give notice periods and so on across the network.
This will unfortunately be a manual process and our tea boy has been dragged in on this (and whilst I am here ... tea anyone? :D ) with a team of administrators who are aiding us. We are not going to be able to get development time to build/modify any databases for this so that idea is currently not an option (hence us trying a manual approach).
The alternative is that we let things carry on? However, we are unwilling to do this. We are only as good as the data that we receive and so we are trying to improve that and nip issues in the bud.
Even merchants with millions of products will only have 100s of categories at worst, so why not manually map their categories to the most relevant SW category? This is the most logical approach and problems would only occur for products that were categorised incorrectly on the merchant end anyway.
This currently is the case. Categories can be manually mapped (which is what I did for the merchants listed in the other thread).
However, the problem arises when the merchant has mapped the categories in a problematic way.
For example:
A merchant has a category in their datafeed called HOME ENTERTAINMENT. HOME ENTERTAINMENT is then mapped to HOME ENTERTAINMENT in our category tree, however, their HOME ENTERTAINMENT category includes their gaming consoles.
This then means that their gaming consoles show up in HOME ENTERTAINMENT (which, if you take a look at this category and its descendants is incorrectly placed).
How do we resolve this? Can we map the category to CONSOLES? No, because then the DVD Players and Projectors would appear under CONSOLES. So in this situation mapping it to HOME ENTERTAINMENT is the best of a bad situation. We cannot "split" that category so that the consoles can be mapped to CONSOLES. The change would have to come in the merchant datafeed where they create a new category for the consoles and then this category can consequently be mapped to CONSOLES.
Dose that make sense? :confused: I am confusing myself now.
That makes perfect sense George. I see the problem now. (Although it still doesn't explain bread knives in the home insurance category).
So is the plan to use the product examples we submit to manually correct each product, or to build a keyword-based tool to re-categorise specific sets of products? e.g. "wii" -> consoles, "playstation" -> consoles
I realise that's not a fool proof option either but it could provide a good first pass before the manual checks take place.
Confuscius
19-11-08, 12:05
There is another way to potentially address this issue that would not involve banning a merchant for large scale miscategorisation that from a development point of view would not be too difficult to implement, as they would be secondary routines to existing processes!
The basic outline would be as follows. As each Merchant feed is read in then a table of all categories used by each merchant would be generated so we end up with a global Control Table with a set of table rows such as:
Merchant A Category 111
Merchant A Category 222
Merchant A Category 333
Merchant B Category 222
Merchant B Category 444
Default set up is that all categories are assumed to be properly mapped. Let's say I examine Merchant B Category 444 and find that some of the products should be mapped to Category 333 - this issue cannot be fixed unless you identify each Product that is wrongly categorised and map it at product level to Category 333. So Merchant B Category 444 should be flagged as an excludable Category and the Merchant requested to build category mapping into THEIR end of the system to resolve the issue or AW continue mapping products forever which is not a very efficient solution. Once product mapping is complete and AW commit to maintaining the data forever then the Category can now be considered to be properly mapped and the flag set to OK! Now all we need to do is collectively review each Merchant - easy to do with a spreadsheet and a couple of pivot tables - we could even create an instruction sheet on how to review a Merchant!
Then, we need to create a global exclusion rule similar to adult content or blocking a merchant - in fact a hybrid of the two and add it to the Merchant Control with an on/off switch. So turn it on and you exclude problem Merchant Categories. Now lots of affiliates start excluding these problem Categories and Merchants might actually want to resolve the issue themselves. Carrots and sticks time.
Just a thought for now - think about it in relation to say the Play feed!
Paul
Hi George,
I’ve just noticed this thread and your related one and I’m pleased that someone at AW is trying to sort this out – I’ve just been blocking the main offenders from my price comparisons, the worst I think being Chemist Direct.
Before I however start submitting reports I have a question. Why can someone at AW not simply click on the retailers under each category and easily see find the offenders. Sort that out and they we affiliates will be easily able to monitor new merchants coming on board.
Go to Photography for example and you can hardly find a camcorder under Camcorders because of batteries from Budget Batteries and accessories from MyMemory, Warehouse Express and Casio. Other offenders are PicStop and PIXmania.
The above offenders are at least displaying camera accessories, PriceRightHome has computer mice, BigRedWareHouse gift sets and Saverstore webcams.
Shrewdie
Hi George,
This is going to be huge task - just looked at Photo Development and WarehouseExpress again is the culprit with loads of accessories listed.
Shrewdie