• DE&I Commitment
  • Careers
  • Locations
  • Book a Meeting
    Book a Meeting
  • Company
    Learn a little more about us, our values, and our team
    Our Values
    We don't just talk the talk; we live by our core values
    About Us
    Empowering brands to realize their potential with data, insights, and technology
    Leadership
    Meet our leadership team
    Secure Data Architecture
    Our promise of data security and privacy. We keep your data safe from publishers, competitors and bad actors
    Corporate Social Responsibility
    We are committed to making a positive impact on our communities and our planet
  • Platform
    Plan, manage, optimize, and measure your campaigns with our omnichannel platform
    Our Platform
    A platform that connects all walled garden media
    Connected Media
    Create and manage campaigns across search, social, retail media and apps, in one platform
    Connected Data
    Make data-driven decisions as you plan and strategize
    See the industries we serve
    Learn how our customizable solutions can help with your unique needs
    Explore our partner integrations
    See the media, retailer, and data partners we work with
  • Clients
  • News and Events
    Check out recent announcements and see what we’re up to
    News
    Check out our recent media coverage
    Events
    Join us for our next conference or webinar
    Quarterly Trends Webinar
    Get the latest insights on digital marketing campaign performance for Q4 2022
    Retail Media for Grocery Webinar
    Join us to dig into trends and challenges shaping the industry in 2023
  • Resources
    From new releases, to industry trends and best practices, Skai has you covered
    Blog
    Read the latest insights and thought leadership from our industry experts
    Capabilities
    Take your campaigns to the next level by enhancing your platform capabilities
    Research
    Explore our reports and whitepapers so you can keep up on the latest industry trends
    Subscribe
    Sign up to get the latest updates straight to your inbox
    Quarterly Trends Report
    Learn digital advertising campaign performance trends from Q3 2022
Back to Blog

Extracting Market Intelligence from Ecommerce Websites: It’s a Jungle Out There

Skai™ Blog

July  13, 2020
Online shopping concept of business woman hand touching shopping cart icon

Online sales are booming. Even before the COVID-19 pandemic, shopping malls were on the decline as more and more consumers and manufacturers turned to the digital world. But analyzing product performance online is much different than in the physical world. There is no universal product code (UPC), many products have different names, and even products with the same name may be different. Each retailer has its own rules, leaving manufacturers scrambling to determine their “share of shelf”, which product reviews to count, what the true rating is for a particular product, how to analyze their competition against their performance, how to optimize their product offering on a particular ecommerce channel and more. In short, it’s a jungle out there.

The challenge of extracting ecommerce market intelligence from websites

Ecommerce sites behave very differently from brick-and-mortar. With so many open marketplaces, such as Amazon, Walmart, e-Bay and Target, and with low barriers to entry for new merchants, the amount of inventory that is uploaded and sold online is staggering. As buyers, being able to find the product you want and to compare it across different channels or even within the same channel can be a real head-scratcher.

Imagine searching for that perfect pink lipstick and seeing the results below. Are these the same lipsticks? If not – how are they different and what’s the best deal for my needs?

But it’s not only the buyers who are confused. Consider brands that use multiple third-party distributors and are trying to track how their products and their competitors are performing online.

Extracting ecommerce market intelligence with advanced analytics

The ecommerce giants are not in a position to solve this issue, as it is in their interest to attract more merchants and expand their reach. As long as there is a low barrier to entry for new merchants who bring in more inventory and create more competition, buyers will continue to purchase and, as a result, tracking and understanding conversion rates for product groupings will continue to be difficult.

The reality is, creating standardization and enforcing sellers to comply with naming conventions, providing universal identifiers such as UPC and mapping inventory to the correct category is not an attractive proposition. It is up to brands to figure it out on their own. One of the newest techniques at their disposal is a Natural Language Processing technology incorporated in the Skai advanced analytics platform, called product clustering.

The product clustering capability groups together all the different listings of a product into one view, across all different merchants and distribution channels. Under this consolidated view, it is possible to have a singular understanding of how a product is performing, for example, how many reviews has it generated and from which channels, what are the consumers saying about the product, and what is its average rating. Moreover, it is possible to benchmark a product against an entire portfolio and compare it against the competition.

Because not every brand defines product clusters in the same way, it is also necessary for advanced analytics platforms to allow flexibility in how the groupings are recognized. Consider two different flavors of your favorite potato chips brand. They come in the same bag, the same size, they’re even sold next to each other on the shelf. Are they unique on their own or should they be clustered as one product line? And what if the same flavor comes in different bag sizes? Or sold individually, in a pack of 3, or a pack of 6? The truth is always in the eye of the beholder.

The secret sauce: NLP breakthroughs for extracting market intelligence from ecommerce websites

Applying a quality product clustering solution on top of 100s of thousands of products from multiple e-commerce channels is not a trivial task. To achieve product clustering with high degrees of accuracy, Skai employs a unique combination of processes that utilize patented NLP technologies, highly scalable autoML capabilities, and brand refinement machine learning algorithms.

It starts with the ability to extract deep knowledge about a product that is ingested into the platform and structures this knowledge into proprietary data models. This includes identifying the product’s solution type and other key features and benefits as well as many other product attributes and then normalizing and refining all of the brand values that are identified to ensure naming consistency within the entire data set.

After the data is organized, it gets segmented. This step determines the criteria that belong to a certain cluster. For example, only products that have the same brand name will be part of a specific segment. The segment can be further winnowed down if only products of the same solution type, flavor, color, size, etc. are included.

Next, dictionaries are generated, curated, and applied to the data which effectively cleans the titles of messy keyword stuffing and other nonrelevant terms. These two steps combined are performed by expert analysts and with autoML capabilities; there is no additional coding work required to create these unique configurations.

Finally, after the data has been organized, segmented, and cleaned, a K-means algorithm scans the full data set and determines the optimal clustering arrangement per segment. This algorithm is manually evaluated and measured for precision (homogeneity of the cluster) and recall (completeness of the cluster). This approach has been shown to yield greater than 95% accuracy, which is well above the industry standard of around 78%.

————————————–

*This blog post originally appeared on Signals-Analytics.com. Kenshoo acquired Signals-Analytics in December 2020. Read the press release.

Request a Demo of Skai

Related Posts

  • Analyzing metrics and charts
    Market Intelligence vs. Market Research: What’s the difference?
    Read More
  • Skai Named “Best Overall AI-based Analytics Company” in 2021 Artificial Intelligence Breakthrough Awards Program 
    Read More
  • Woman looking and analyzing her computer screen while holding a pen in her hand.
    What is Modern Market Intelligence?
    Read More
  • Multiple networks and connections
    6 Insights You Should Get from Your Market Intelligence Data
    Read More
  • Post-Pandemic Boom: Advertisers Spent 3.7X during Prime Day 2021 to Boost Sales 
    Read More
  • Skai’s Chief product officer guy cohen on market intelligenceSkai’s Chief product officer guy cohen on market intelligence
    The Skai 5: Five Questions About Market Intelligence With Skai’s Guy Cohen, Chief Product Officer 
    Read More
  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share via Email
  • Copy Link
    Copied!

Subscribe to Updates

Media that matters.
Marketing that works.
© 2023 Kenshoo, Ltd. All Rights Reserved.
Privacy Policy. Cookie Policy. Recruitment Privacy Policy.
  • Connected Data
    • Market Intelligence
    • Our Approach
    • By Need
    • By Solution
  • Connected Strategy
    • Dynamic Marketing Mix
    • Budget Forecasting
    • Strategic Consulting
  • Connected Media
    • Overview
    • Retail Media
    • Paid Search
    • Paid Social
    • App Marketing
    • Auditing
    • Expert Services
  • Measurement
    • Incrementality
    • Experiments
    • Cross-Channel Attribution
  • Resources
    • Blog
    • Glossary
    • Case Studies
    • Training & Enablement
    • Developer Hub
Privacy Preference

We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.

Privacy Preference

Save All

Save

Accept Only Essential Cookies

Manage Cookie Preferences

Cookie Details Privacy Policy Imprint

Privacy Preference

Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

Save All Save Accept Only Essential Cookies

Back

Privacy Preference

Essential cookies enable basic functions and are necessary for the proper function of the website.

Show Cookie Information Hide Cookie Information

Name
Provider Owner of this website, Imprint
Purpose Saves the visitors preferences selected in the Cookie Box of Borlabs Cookie.
Host(s) .skai.io, skai.io
Cookie Name borlabs-cookie
Cookie Expiry 1 Year
Name
Provider Owner of this website
Purpose This cookie stores selections made by the user in the Accessibe tool in order to maintain those settings on future visits. These cookies help us make our website compliant with our obligations under US law.
Privacy Policy https://accessibe.com/privacy-policy
Cookie Name acsbState, acsbReset
Cookie Expiry n/a
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name wordpress_sec_,wordpress_test_cookie,wp-postpass_*, wordpresspass_*, wordpressuser_*
Cookie Expiry Session / 1 Year

We use these cookies to enhance functionality and allow for personalisation, such as live chats, videos and the use of social media.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Owner of this website
Host(s) .chilipiper.com, skai.chilipiper.com
Cookie Name fs_uid, CHILI_PIPER_CLUSTER, guest-session, _sp_ses*, _sp_id*
Cookie Expiry Session / 2 Years
Accept
Name
Provider Owner of this website
Host(s) .comeet.co, www.comeet.co
Cookie Name visid_incap_, nlbi_#######, incap_ses_, referrer22_00a, incap_ses_1364_2167377
Cookie Expiry Session / 1 Year
Accept
Name
Provider Owner of this website
Host(s) skai.io
Cookie Name moduleFormPardotDownload
Cookie Expiry 30 days

Statistics cookies collect information anonymously. This information helps us to understand how our visitors use our website.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for website analytics. Generates statistical data on how the visitor uses the website.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name _ga,_ga_*,_gat,_gat_*,_gid
Cookie Expiry 2 Months
Accept
Name
Provider Hotjar Ltd., Dragonara Business Centre, 5th Floor, Dragonara Road, Paceville St Julian's STJ 3141 Malta
Purpose Hotjar is an user behavior analytic tool by Hotjar Ltd.. We use Hotjar to understand how users interact with our website.
Privacy Policy https://www.hotjar.com/legal/policies/privacy/
Host(s) *.hotjar.com
Cookie Name _hjClosedSurveyInvites, _hjDonePolls, _hjMinimizedPolls, _hjDoneTestersWidgets, _hjIncludedInSample, _hjShownFeedbackMessage, _hjid, _hjRecordingLastActivity, hjTLDTest, _hjUserAttributesHash, _hjCachedUserAttributes, _hjLocalStorageTest, _hjptid, _hjSessionUser_2229986, _hjIncludedInPageviewSample, _hjIncludedInSessionSample, _hjAbsoluteSessionInProgress, _hjFirstSeen
Cookie Expiry Session / 1 Year

Marketing cookies are used by third-party advertisers or publishers to display personalized ads. They do this by tracking visitors across websites.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Linkedin
Cookie Name lidc, li_gc, lang, AnalyticsSyncHistory, UserMatchHistory, li_sugr, bcookie, TDCPM, TDID, bscookie, ln_or
Cookie Expiry Session / 1 Year
Accept
Name
Provider Skai
Accept
Name
Provider 6sense
Cookie Name _gd_session, _an_uid, _gd_visitor, _gd_svisitor, 6suuid
Cookie Expiry Session / 400 Days
Accept
Name
Provider Pardot
Purpose Cookie name associated with services from marketing automation and lead generation platform Pardot. The visitor value is the visitor_id in your Pardot account. This cookie is set for visitors by the Pardot tracking code.
Host(s) .pardot.com, pi.pardot.com, skai.io
Cookie Name pardot, visitor_id*, lpv*
Cookie Expiry Session / 10 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Cookie by Google used for conversion tracking of Google Ads.
Privacy Policy https://policies.google.com/privacy?hl=en
Cookie Name IDE, 1P_JAR, NID, SOCS, CONSENT, AEC, _gcl_au, OTZ, test_cookie
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Cookie by Facebook used for website analytics, ad targeting, and ad measurement.
Privacy Policy https://www.facebook.com/policies/cookies
Cookie Name _fbp,act,c_user,datr,fr,tr,m_pixel_ration,pl,presence,sb,spin,wd,xs
Cookie Expiry Session / 1 Year

Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

Show Cookie Information Hide Cookie Information

Accept
Name
Provider Wistia
Host(s) .wistia.com
Cookie Name cb_anonymous_id, _sp_ses.2b40, _li_dcdm_c, __hssrc, _gcl_au, _clsk, hubspotutk, _sp_id.2b40, __hssc, __hstc, _uetsid, _uetvid, _gid, _ga, _ga_GQR109DZ3Y, _lc2, fpi, _ex-pricing-cta, _fbp, cb_group_id, cb_user_id, _clck
Cookie Expiry Session / 400 Days
Accept
Name
Provider Meta Platforms Ireland Limited, 4 Grand Canal Square, Dublin 2, Ireland
Purpose Used to unblock Instagram content.
Privacy Policy https://www.instagram.com/legal/privacy/
Host(s) .instagram.com
Cookie Name pigeon_state
Cookie Expiry Session
Accept
Name
Provider Openstreetmap Foundation, St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS, United Kingdom
Purpose Used to unblock OpenStreetMap content.
Privacy Policy https://wiki.osmfoundation.org/wiki/Privacy_Policy
Host(s) .openstreetmap.org
Cookie Name _osm_location, _osm_session, _osm_totp_token, _osm_welcome, _pk_id., _pk_ref., _pk_ses., qos_token
Cookie Expiry 1-10 Years
Accept
Name
Provider Twitter International Company, One Cumberland Place, Fenian Street, Dublin 2, D02 AX07, Ireland
Purpose Used to unblock Twitter content.
Privacy Policy https://twitter.com/privacy
Host(s) .twimg.com, .twitter.com
Cookie Name __widgetsettings, local_storage_support_test
Cookie Expiry Unlimited
Accept
Name
Provider Vimeo Inc., 555 West 18th Street, New York, New York 10011, USA
Purpose Used to unblock Vimeo content.
Privacy Policy https://vimeo.com/privacy
Host(s) player.vimeo.com
Cookie Name vuid
Cookie Expiry 2 Years
Accept
Name
Provider Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Ireland
Purpose Used to unblock YouTube content.
Privacy Policy https://policies.google.com/privacy?hl=en&gl=en
Host(s) google.com
Cookie Name CONSENT
Cookie Expiry 6 Month

Borlabs Cookie powered by Borlabs Cookie

Privacy Policy Imprint