Global AI and Data Science

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.


To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.



------------------------------
David W. Kovalcik
------------------------------
#GlobalAIandDataScience
#GlobalDataScience

Hello Watson Studio Community,

I am, Shilpee Shrestha from the Marketplace Governance team mentioned in the Dave's discussion above. I would like to add a couple more questions


------------------------------
SHILPEE SHRESTHA
------------------------------

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------
#GlobalAIandDataScience
#GlobalDataScience

>QUESTION #5: Once we create our model and class in a project, how do we deploy it to production. We do not find the option as illustrated in the documentation guide. Is it because we are on free-trial?

Once you train a model, it is automatically deployed.  In Studio, you click the "test" button.  In the API, you use the /classify endpoint, as described here https://cloud.ibm.com/docs/services/visual-recognition?topic=visual-recognition-tutorial-custom-classifier#tutorial-custom-classifier-classify

>QUESTION #6: Can we add more images to the same class once we train it already?

Yes, we call this retraining.  You can add new images to existing classes, or new images to new classes within the existing classifier.  In Studio, just click "edit and retrain" button. In the API, use POST v3/classifiers/{classifier_id_to_retrain}  for details see https://cloud.ibm.com/docs/services/visual-recognition?topic=visual-recognition-tutorial-custom-classifier#tutorial-custom-classifier-update



------------------------------
Matt Hill
------------------------------

Original Message:
Sent: Wed February 12, 2020 02:28 PM
From: SHILPEE SHRESTHA
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community,

I am, Shilpee Shrestha from the Marketplace Governance team mentioned in the Dave's discussion above. I would like to add a couple more questions

QUESTION #5: Once we create our model and class in a project, how do we deploy it to production. We do not find the option as illustrated in the documentation guide. Is it because we are on free-trial?

QUESTION #6: Can we add more images to the same class once we train it already?

------------------------------
SHILPEE SHRESTHA

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------
#GlobalAIandDataScience
#GlobalDataScience

Hello Watson Studio Community,

I am Shilpee Shrestha, from the Marketplace Governance Team mentioned in the Dave's original post. I would like to add a couple more questions:


------------------------------
SHILPEE SHRESTHA
------------------------------

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David W. Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an analytics solution that evaluates the context of each item and recommends the most appropriate classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Watson Visual Recognition proof-of-concept is focused on determining whether we can significantly improve the confidence scores while also validating proper image assignment to marketplace items.

Issues / Questions / Help Needed
Within our Visual Recognition Project, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------

#GlobalAIandDataScience
#GlobalDataScience

------------------------------
Matt Hill
------------------------------

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------
#GlobalAIandDataScience
#GlobalDataScience

I came here to ask the same thing.

------------------------------
Gregory O'Shea
------------------------------

Original Message:
Sent: Thu February 13, 2020 03:35 PM
From: Matt Hill
Subject: Watson Visual Recognition Models & Classes

>QUESTION #1:  Is this limit due to our use of a "free trial" account?

Yes, it appears so.  Lite plan accounts are limited to 1000 images per month, including training and testing.  If you upgrade to standard, the work you have already done will still be in the account.

>QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

1000 events would be 1000 images per month, whether used in training or classification.  When retraining an existing model, only the new images added are counted towards this for that iteration.  See https://www.ibm.com/cloud/watson-visual-recognition/pricing for details (scroll down to "Pricing details")

>QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

If you're using the API, the 250MB limit applies to each request.  So you can iteratively train a model with arbitrarily large size by submitting batches of 250MB of training data.  Note: for classification, you can resize your images to 224x224 pixels before sending them without any loss.  Using Watson Studio, you may be limited.  We have an open issue in Studio investigating this, dap/dap-planning/issues/9829 if you're an IBMer.

>QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

Creating that many models could cause some issues, particularly in Studio.  It might be better to use a hierarchical system, where you have 200 models of 100 classes each and 100-class classifier as a first tier.  However, this can be a bit brittle.  Having 18000 classes in a single classifier is not a problem for the API, if built iteratively via retraining, as mentioned above.  The general tagging model exceeds that size.

------------------------------
Matt Hill

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------
#GlobalAIandDataScience
#GlobalDataScience

@Matt Hill, thank you for taking the time to share your expertise with us.  Greatly appreciated!!!

If I am following the gist of your responses, it seems that our use of Watson Studio vs the APIs is the root of our immediate issues.  For clarification, I have attached a PPT containing screenshots of the various error we are receiving.  

We are now getting errors each time we try to train a new class or retrain an existing class.  I assume this is due to the 1000 monthly limit on images.  Thus, our POC is essentially stuck at the moment.

Additional Questions/Clarifications

1. You indicated that we could resize images to 224 x 224 pixels with no-loss.  Surely this is impacted by the quality (dpi) of the original images.  Do you know if there is any published documentation that addresses the recommended image characteristics? 
   
   
2. Regarding your response to Question #3:  See page 5 of the attached PPT.  Watson Studio seems to be limiting our entire VR model to 250 MB.  The error states: "You have exceeded 250 MB of training data. To begin training, you must unclassify or delete 36.3 MB of data."  Based upon your response, am I correct in understanding that this limit is enforced by Watson Studio but not with the API?
   
   
3. Regarding your response to Question #5:  Are you saying that each time we "test" a classification within Watson Studio, that test data and classification is assimilated into the overall Watson VR data corpus?  If so, we're not seeing these results when we go back and test the existing Watson VR models.  If we load one of our test/training images to the Watson VR general model, the image is not classified with our newly trained classification.

@SHILPEE SHRESTHA & @Vikram Pratap Singh see above and add any additional thoughts.

------------------------------
David Kovalcik P2P Solutions & Technology Lead, IPS - Delivery
------------------------------

Original Message:
Sent: Thu February 13, 2020 03:35 PM
From: Matt Hill
Subject: Watson Visual Recognition Models & Classes

>QUESTION #1:  Is this limit due to our use of a "free trial" account?

Yes, it appears so.  Lite plan accounts are limited to 1000 images per month, including training and testing.  If you upgrade to standard, the work you have already done will still be in the account.

>QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

1000 events would be 1000 images per month, whether used in training or classification.  When retraining an existing model, only the new images added are counted towards this for that iteration.  See https://www.ibm.com/cloud/watson-visual-recognition/pricing for details (scroll down to "Pricing details")

>QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

If you're using the API, the 250MB limit applies to each request.  So you can iteratively train a model with arbitrarily large size by submitting batches of 250MB of training data.  Note: for classification, you can resize your images to 224x224 pixels before sending them without any loss.  Using Watson Studio, you may be limited.  We have an open issue in Studio investigating this, dap/dap-planning/issues/9829 if you're an IBMer.

>QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

Creating that many models could cause some issues, particularly in Studio.  It might be better to use a hierarchical system, where you have 200 models of 100 classes each and 100-class classifier as a first tier.  However, this can be a bit brittle.  Having 18000 classes in a single classifier is not a problem for the API, if built iteratively via retraining, as mentioned above.  The general tagging model exceeds that size.

------------------------------
Matt Hill

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------​​​
#GlobalAIandDataScience
#GlobalDataScience

>We are now getting errors each time we try to train a new class or retrain an existing class.  I assume this is due to the 1000 monthly limit on images.  Thus, our POC is essentially stuck at the moment.

Yes, you should upgrade to a standard plan if you want to do more than the 1000 images per month the Lite plan offers.  IBM internal billing gets a substantial discount, and you don't need a credit card or anything.  Many POCs can be done with < 1000 images, but our Standard plan is quite cheap too.

>You indicated that we could resize images to 224 x 224 pixels with no-loss.  Surely this is impacted by the quality (dpi) of the original images.  Do you know if there is any published documentation that addresses the recommended image characteristics? 

Not really.  The first thing the API does internally is resize the image to 224x224 pixels.  So if you do it first, there is no impact.  If the thing you want to classify cannot be seen by a human at 224^2, you'll need to zoom in, tile or otherwise acquire images in which it can be.  Docs on this are here, our main docs:  https://cloud.ibm.com/docs/services/visual-recognition?topic=visual-recognition-customizing#customizing-guidelines-training

>Regarding your response to Question #3:  See page 5 of the attached PPT.  Watson Studio seems to be limiting our entire VR model to 250 MB.  The error states: "You have exceeded 250 MB of training data. To begin training, you must unclassify or delete 36.3 MB of data."  Based upon your response, am I correct in understanding that this limit is enforced by Watson Studio but not with the API?

Your initial training set, and each additional block of more training data, has to be under 250MB.  The API does not accept a single request >250MB.

> Regarding your response to Question #5:  Are you saying that each time we "test" a classification within Watson Studio, that test data and classification is assimilated into the overall Watson VR data corpus? 

No, not saying that.  Using the "test" button or the v3/classify API endpoint simply applies a model to the input image and gives a classification.  It does not change the model at all.  It's like a function which applies a model to the image and gives a result, like in pseudocode:
`classify(myModel_1234, myimage,jpg) ==>  "dog, 0.65"`

The input here is a model and an image.  To train or retrain, you need to know the "truth" of what the image is -- the user-given label.  So a training example needs to be a pair of some image and some metadata label.





------------------------------
Matt Hill
------------------------------

Original Message:
Sent: Mon February 17, 2020 12:02 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

@Matt Hill, thank you for taking the time to share your expertise with us.  Greatly appreciated!!!

If I am following the gist of your responses, it seems that our use of Watson Studio vs the APIs is the root of our immediate issues.  For clarification, I have attached a PPT containing screenshots of the various error we are receiving.  

We are now getting errors each time we try to train a new class or retrain an existing class.  I assume this is due to the 1000 monthly limit on images.  Thus, our POC is essentially stuck at the moment.

Additional Questions/Clarifications

1. You indicated that we could resize images to 224 x 224 pixels with no-loss.  Surely this is impacted by the quality (dpi) of the original images.  Do you know if there is any published documentation that addresses the recommended image characteristics? 
   
   
2. Regarding your response to Question #3:  See page 5 of the attached PPT.  Watson Studio seems to be limiting our entire VR model to 250 MB.  The error states: "You have exceeded 250 MB of training data. To begin training, you must unclassify or delete 36.3 MB of data."  Based upon your response, am I correct in understanding that this limit is enforced by Watson Studio but not with the API?
   
   
3. Regarding your response to Question #5:  Are you saying that each time we "test" a classification within Watson Studio, that test data and classification is assimilated into the overall Watson VR data corpus?  If so, we're not seeing these results when we go back and test the existing Watson VR models.  If we load one of our test/training images to the Watson VR general model, the image is not classified with our newly trained classification.

@SHILPEE SHRESTHA & @Vikram Pratap Singh see above and add any additional thoughts.

------------------------------
David Kovalcik P2P Solutions & Technology Lead, IPS - Delivery

Original Message:
Sent: Thu February 13, 2020 03:35 PM
From: Matt Hill
Subject: Watson Visual Recognition Models & Classes

>QUESTION #1:  Is this limit due to our use of a "free trial" account?

Yes, it appears so.  Lite plan accounts are limited to 1000 images per month, including training and testing.  If you upgrade to standard, the work you have already done will still be in the account.

>QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

1000 events would be 1000 images per month, whether used in training or classification.  When retraining an existing model, only the new images added are counted towards this for that iteration.  See https://www.ibm.com/cloud/watson-visual-recognition/pricing for details (scroll down to "Pricing details")

>QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

If you're using the API, the 250MB limit applies to each request.  So you can iteratively train a model with arbitrarily large size by submitting batches of 250MB of training data.  Note: for classification, you can resize your images to 224x224 pixels before sending them without any loss.  Using Watson Studio, you may be limited.  We have an open issue in Studio investigating this, dap/dap-planning/issues/9829 if you're an IBMer.

>QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

Creating that many models could cause some issues, particularly in Studio.  It might be better to use a hierarchical system, where you have 200 models of 100 classes each and 100-class classifier as a first tier.  However, this can be a bit brittle.  Having 18000 classes in a single classifier is not a problem for the API, if built iteratively via retraining, as mentioned above.  The general tagging model exceeds that size.

------------------------------
Matt Hill

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------​​​
#GlobalAIandDataScience
#GlobalDataScience

@Matt Hill, thank you for the detailed and extremely helpful response!

------------------------------
David Kovalcik P2P Solutions & Technology Lead, IPS - Delivery
------------------------------

Original Message:
Sent: Wed February 19, 2020 12:15 PM
From: Matt Hill
Subject: Watson Visual Recognition Models & Classes

>We are now getting errors each time we try to train a new class or retrain an existing class.  I assume this is due to the 1000 monthly limit on images.  Thus, our POC is essentially stuck at the moment.

Yes, you should upgrade to a standard plan if you want to do more than the 1000 images per month the Lite plan offers.  IBM internal billing gets a substantial discount, and you don't need a credit card or anything.  Many POCs can be done with < 1000 images, but our Standard plan is quite cheap too.

>You indicated that we could resize images to 224 x 224 pixels with no-loss.  Surely this is impacted by the quality (dpi) of the original images.  Do you know if there is any published documentation that addresses the recommended image characteristics? 

Not really.  The first thing the API does internally is resize the image to 224x224 pixels.  So if you do it first, there is no impact.  If the thing you want to classify cannot be seen by a human at 224^2, you'll need to zoom in, tile or otherwise acquire images in which it can be.  Docs on this are here, our main docs:  https://cloud.ibm.com/docs/services/visual-recognition?topic=visual-recognition-customizing#customizing-guidelines-training

>Regarding your response to Question #3:  See page 5 of the attached PPT.  Watson Studio seems to be limiting our entire VR model to 250 MB.  The error states: "You have exceeded 250 MB of training data. To begin training, you must unclassify or delete 36.3 MB of data."  Based upon your response, am I correct in understanding that this limit is enforced by Watson Studio but not with the API?

Your initial training set, and each additional block of more training data, has to be under 250MB.  The API does not accept a single request >250MB.

> Regarding your response to Question #5:  Are you saying that each time we "test" a classification within Watson Studio, that test data and classification is assimilated into the overall Watson VR data corpus? 

No, not saying that.  Using the "test" button or the v3/classify API endpoint simply applies a model to the input image and gives a classification.  It does not change the model at all.  It's like a function which applies a model to the image and gives a result, like in pseudocode:
`classify(myModel_1234, myimage,jpg) ==>  "dog, 0.65"`

The input here is a model and an image.  To train or retrain, you need to know the "truth" of what the image is -- the user-given label.  So a training example needs to be a pair of some image and some metadata label.





------------------------------
Matt Hill

Original Message:
Sent: Mon February 17, 2020 12:02 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

@Matt Hill, thank you for taking the time to share your expertise with us.  Greatly appreciated!!!

If I am following the gist of your responses, it seems that our use of Watson Studio vs the APIs is the root of our immediate issues.  For clarification, I have attached a PPT containing screenshots of the various error we are receiving.  

We are now getting errors each time we try to train a new class or retrain an existing class.  I assume this is due to the 1000 monthly limit on images.  Thus, our POC is essentially stuck at the moment.

Additional Questions/Clarifications

1. You indicated that we could resize images to 224 x 224 pixels with no-loss.  Surely this is impacted by the quality (dpi) of the original images.  Do you know if there is any published documentation that addresses the recommended image characteristics? 
   
   
2. Regarding your response to Question #3:  See page 5 of the attached PPT.  Watson Studio seems to be limiting our entire VR model to 250 MB.  The error states: "You have exceeded 250 MB of training data. To begin training, you must unclassify or delete 36.3 MB of data."  Based upon your response, am I correct in understanding that this limit is enforced by Watson Studio but not with the API?
   
   
3. Regarding your response to Question #5:  Are you saying that each time we "test" a classification within Watson Studio, that test data and classification is assimilated into the overall Watson VR data corpus?  If so, we're not seeing these results when we go back and test the existing Watson VR models.  If we load one of our test/training images to the Watson VR general model, the image is not classified with our newly trained classification.

@SHILPEE SHRESTHA & @Vikram Pratap Singh see above and add any additional thoughts.

------------------------------
David Kovalcik P2P Solutions & Technology Lead, IPS - Delivery

Original Message:
Sent: Thu February 13, 2020 03:35 PM
From: Matt Hill
Subject: Watson Visual Recognition Models & Classes

>QUESTION #1:  Is this limit due to our use of a "free trial" account?

Yes, it appears so.  Lite plan accounts are limited to 1000 images per month, including training and testing.  If you upgrade to standard, the work you have already done will still be in the account.

>QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

1000 events would be 1000 images per month, whether used in training or classification.  When retraining an existing model, only the new images added are counted towards this for that iteration.  See https://www.ibm.com/cloud/watson-visual-recognition/pricing for details (scroll down to "Pricing details")

>QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

If you're using the API, the 250MB limit applies to each request.  So you can iteratively train a model with arbitrarily large size by submitting batches of 250MB of training data.  Note: for classification, you can resize your images to 224x224 pixels before sending them without any loss.  Using Watson Studio, you may be limited.  We have an open issue in Studio investigating this, dap/dap-planning/issues/9829 if you're an IBMer.

>QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

Creating that many models could cause some issues, particularly in Studio.  It might be better to use a hierarchical system, where you have 200 models of 100 classes each and 100-class classifier as a first tier.  However, this can be a bit brittle.  Having 18000 classes in a single classifier is not a problem for the API, if built iteratively via retraining, as mentioned above.  The general tagging model exceeds that size.

------------------------------
Matt Hill

Original Message:
Sent: Tue February 11, 2020 02:49 PM
From: David Kovalcik
Subject: Watson Visual Recognition Models & Classes

Hello Watson Studio Community! 

We are seeking community advice and guidance with our Visual Recognition project within Watson Studio.  Any and all advice and discussion is greatly appreciated.

Initiative Background
I lead a group of people who make up the Governance Team for a new marketplace initiative and we are currently focused on expanding the Marketplace's business value proposition through the incorporation of AI and Machine Learning.  We have kicked-off a Visual Recognition proof-of-concept project within Watson Studio with the explicit purpose of determining whether VR can successfully address an ongoing challenge for all marketplaces:  proper item/product classification. 

Business Problem
Proper classification of marketplace items is essential for usability as well as the proper functioning of many downstream features and processes.  As sellers/suppliers load their products and services to the marketplace, they are required to classify each item with the most precise classification code possible (UNSPSC.org).  Unfortunately, proper classification by suppliers is notoriously poor for a multitude of reasons including:

• Use of old and/or incorrect classification versions
• Copy/Paste mistakes
• Lack of effort, experience/skills, and/or resource bandwidth

Proposed Solution
Using various technologies, we have incorporated an "Automated Classification Analysis Tool" that is an analytics solution that evaluates the context of each item and recommends the most appropriate UNSPSC classification code(s).  Unfortunately, many marketplace items do not have ample context to ensure a highly reliable confidence score in the recommended classification(s).

This Proof-of-Concept project is focused on determining whether we can significantly improve the results and confidence scores of the "Automated Classification Analysis Tool" by not only evaluating the context of marketplace items but by incorporating Watson VR to also analyze their associated images.

EXAMPLE:  if a seller/supplier loads a computer mouse to the marketplace, Watson must evaluate the associated images that the supplier provided. If Watson cannot verify that each image represents a computer mouse, the item will be flagged for manual review by a member of the Marketplace Governance Team.  If Watson is correct, the supplier will be asked to correct the item by providing valid images. If Watson is incorrect, the Governance Team will utilize the images to further train Watson VR.

To train Watson VR on each of the targeted classification codes (~18K), we will initially load and train Watson with 100 – 500 images per UNSPSC classification code.

Issues / Questions / Help Needed
Within our Visual Recognition Project within Watson Studio, we have created a VR model and began creating classifications for marketplace categories (UNSPSC.org codes) and loading training images for each.  When loading the images, we are receiving an error stating that the training data must not exceed 250MB.

QUESTION #1:  Is this limit due to our use of a "free trial" account?

QUESTION #2:  Is this limit per classification or for the entire model or just the newly added training data?

QUESTION #3:  The marketplace will eventually consist of ~18K classifications and we will provide ~100 training images for each.  If the 250MB limit is not due to our use of a trail-account, that would seem to indicate that we will have to spread the ~18K marketplace classifications across many VR models.  Is this accurate?

QUESTION #4:  Based on our training and understanding, the creation of thousands of VR models does not seem appropriate for our purposes.  Any suggestions/recommendations?

------------------------------
David W. Kovalcik
------------------------------​​​
#GlobalAIandDataScience
#GlobalDataScience