Парсер товаров интернет — магазина Bed Bath & Beyond

Bed Bath & Beyond — сеть магазинов товаров для дома в США, Пуэрто-Рико, Канаде и Мексике. В 1971 году Уоррен Эйленберг и Леонард Фейнштейн открыли магазин с названием Bed ‘n Bath в Спрингфилде, штат Нью Джерси. К 1985 они управляли уже 17 магазинами в Нью Йорке и Калифорнии. Чтобы соответствовать росту, компания переименовалась в Bed Bath & Beyond. Этот парсер предназначен для сбора информации о товарах в интернет-магазине bedbathandbeyond.com.

Парсер обновлен 10 августа 2019 в связи с изменением части фреймворка на сайте

Примерное количество товаров: 200000
Примерное количество запросов: 400000
Рекомендуемый план подписки: Medium

ВНИМАНИЕ! Количество запросов может превышать количество товаров, потому что данные о вариациях, изображениях и др. могут парсится используя запросы к дополнительным ресурсам. Также часть данных о товаре может доставляться с помощью XHR запросов, что также увеличивает общее количество необходимых запросов.

Для его использования вы должны иметь учетную запись в нашем сервисе Diggernaut.

1. Пройдите по этой ссылке для регистрации в сервисе Diggernaut
2. После регистрации и подтверждения email адреса войдите в свою учетную запись
3. Создайте проект с любый именем и описанием, если вы не знаете как, обратитесь к нашей документации
4. Войдите во вновь созданный проект и создайте в нем диггер с любым именем, если вы не знаете как, обратитесь к нашей документации
5. Скопируйте в буфер обмена приведенный ниже сценарий диггера и вставьте его в созданный вами диггер, если вы не знаете как, обратитесь к нашей документации
6. ВНИМАНИЕ! Базовые прокси сервера могут не работать с данным сайтом и вам могут потребоваться ваши собственные прокси. Вы должны будете прописать ваш прокси сервер в сценарии, как указано. Если этот пункт вам непонятен и вы затрудняетесь как это сделать, свяжитесь с нами через систему поддержки или используя наш онлайн чат, мы обязательно вам поможем.
7. Переключите режим работы диггера с Debug на Active, если вы не знаете как, обратитесь к нашей документации
8. Запустите ваш диггер и дождитесь окончания его работы, если вы не знаете как, обратитесь к нашей документации
9. Скачайте собранный набор данных в нужном вам формате, если вы не знаете как, обратитесь к нашей документации

В дальнейшем вы можете установить расписание для запуска вашего парсера и забирать информацию регулярно.

Сценарий парсера:

---
config:
    debug: 2
    agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
    proxy: #USE YOUR PROXY HERE LIKE 1.1.1.1:8888
do:
- variable_set:
    field: repeatcat
    value: "yes"
- variable_set:
    field: repeatitems
    value: "yes"
## --------------------
## categories collector
- walk:
    to: https://www.bedbathandbeyond.com/apis/stateless/v1.0/navigation/category-navigation
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-encoding: deflate
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
        pragma: no-cache
        upgrade-insecure-requests: 1
    do:
    - find:
        path: data:haschild(label:contains("Product")) > menu > items
        slice: 0
        do:
        - node_remove: promo
        - find:
            path: url:contains("category")
            do:
            - parse:
                filter: ^([^\?]+)
            - normalize:
                routine: url
            - if:
                match: \/store\/category\/
                do:
                - normalize:
                    routine: replace_substring
                    args:
                        - \/?$: '/1-96'
                - link_add:
                    pool: categories
- walk:
    to: links
    pool: categories
    repeat_in_pool: 
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-encoding: deflate
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
        pragma: no-cache
        upgrade-insecure-requests: 1
    do:
    - find:
        path: title
        do:
        - parse
        - if:
            match: Access Denied
            do:
            - proxy_switch
            else:
            ## removing repeat
            - variable_clear: repeatcat
            - find:
                path: '#ctl00_InvalidRequest'
                in: doc
                do:
                - parse
                - if:
                    match: \S
                    do:
                    - proxy_switch
                    - variable_set:
                        field: repeatcat
                        value: "yes"
            - find:
                path: body
                in: doc
                do:
                ## main logic to gather product item links
                - find:
                    path: div.mt0.tealium-product-grid
                    do:
                    ## collect all item links from page
                    - find:
                        path: 'div.tealium-product-tile > div[class*="ProductTile-"] > a[class*="PrimaryLink_"]'
                        do:
                        - parse:
                            attr: href
                            filter: ^([^\?]+)
                        - normalize:
                            routine: url
                        - link_add:
                            pool: items
                ## and let's try to find next page here
                - find:
                    path: a.Pagination__btnNext
                    do:
                    - parse:
                        attr: aria-disabled
                    - if:
                        match: "true"
                        do:
                        ## next page is not found
                        else:
                        ## found next page
                        ## add new page link into pool
                        - static_get: url
                        - variable_set: url
                        - filter:
                            args: '\/(\d+)\-\d+$'
                        - variable_set: pageid
                        - eval:
                            routine: js
                            body: '(function () {
                                        var cnt = ;
                                        return cnt + 1;
                                    })();'
                        - variable_set: pageid
                        - variable_get: url
                        - normalize:
                            routine: replace_substring
                            args:
                            - \/\d+\-\d+$: '/-96'
                        - link_add:
                            pool: categories
- walk:
    to: links
    pool: items
    repeat_in_pool: 
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-encoding: deflate
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
        pragma: no-cache
        upgrade-insecure-requests: 1
    do:
    - find:
        path: title
        do:
        - parse
        - if:
            match: Access Denied
            do:
            - proxy_switch
            else:
            - find:
                path: html
                in: doc
                do:
                ## removing repeat
                - variable_clear: repeatitems
            - find:
                path: '#ctl00_InvalidRequest'
                in: doc
                do:
                - parse
                - if:
                    match: \S
                    do:
                    - proxy_switch
                    - variable_set:
                        field: repeatitems
                        value: "yes"
    - find:
        path: body
        in: doc
        do:
        - variable_get: repeatitems
        - if:
            match: \S
            else:
            ## save item
            - object_new: product
            - find:
                path: script:contains("window.__INITIAL_STATE")
                do:
                - parse
                - space_dedupe
                - trim
                - filter: 
                    args: 'window\.__INITIAL_STATE__\s+\=\s+(.+)\s*;\s*window.__INITIAL_STATE__.sitespect'
                - normalize:
                    routine: json2xml
                - to_block
                - find:
                    path: body_safe > pdp > productdetails > data
                    slice: 0
                    do:
                    - variable_clear: pid
                    - variable_set:
                        field: brand
                        value: BedBathAndBeyond
                    - eval:
                        routine: js
                        body: '(function () {
                                    var d = new Date();
                                    return d.toISOString();
                                })();'
                    - object_field_set:
                        object: product
                        field: date
                    - static_get: url
                    - object_field_set:
                        object: product
                        field: url
                    - find:
                        path: brand_name
                        do:
                        - parse
                        - space_dedupe
                        - trim
                        - variable_set: brand
                    - variable_get: brand
                    - object_field_set:
                        object: product
                        field: brand
                    - find:
                        path: display_name
                        do:
                        - parse
                        - space_dedupe
                        - trim
                        - object_field_set:
                            object: product
                            field: name
                    - find:
                        path: description
                        do:
                        - parse
                        - space_dedupe
                        - trim
                        - if:
                            match: \w+
                            do:
                            - object_field_set:
                                object: product
                                field: description
                    - find:
                        path: product_id
                        do:
                        - parse:
                            filter: (\d+)
                        - space_dedupe
                        - trim
                        - object_field_set:
                            object: product
                            field: sku
                    - find:
                        path: low_price
                        do:
                        - parse:
                            filter: ([\d\.]+)
                        - object_field_set:
                            object: product
                            type: float
                            field: price
                    - find:
                        path: variations > all_colors
                        do:
                        - find:
                            path: color
                            do:
                            - parse
                            - space_dedupe
                            - trim
                            - if:
                                match: \w+
                                do:
                                - object_field_set:
                                    object: product
                                    joinby: "|"
                                    field: variations
                    - find:
                        path: image_id
                        do:
                        - parse
                        - normalize:
                            routine: replace_substring
                            args:
                                - '^\s*': 'https://b3h2.scene7.com/is/image/BedBathandBeyond/'
                        - variable_set: image_url
                        - register_set: ?scl=1
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
                    - find:
                        path: alt_img
                        do:
                        - split:
                            context: text
                            delimiter: ','
                        - find:
                            path: div.splitted
                            do:
                            - parse
                            - normalize:
                                routine: replace_substring
                                args:
                                    - '^\s*': 'https://b3h2.scene7.com/is/image/BedBathandBeyond/'
                            - variable_set: image_url
                            - register_set: ?scl=1
                            - object_field_set:
                                object: product
                                joinby: "|"
                                field: images
            - find:
                path: 'div#first > ul[class*="Breadcrumbs-"] > li > a'
                slice: 0:-1
                do:
                - parse
                - space_dedupe
                - trim
                - if:
                    match: \w+
                    do:
                    - object_field_set:
                        object: product
                        joinby: "|"
                        field: category
            - object_save:
                name: product

Ниже приведен пример датасета с несколькими товарами в формате JSON (для наглядности). Датасет может быть скачан и как CSV, XLSX, XML, и любой другой текстовый формат используя темплейтный подход.

[{
    "product": {
        "brand": "Dyson",
        "category": "Gifts|Gifts by Category|Unique Gifts",
        "currency": "USD",
        "date": "2017-12-07T00:05:23.532Z",
        "description": "Dyson's Supersonic Hair Dryer uses intelligent heat control technology to help to prevent heat damage to your hair, preserving its natural shine. This high-speed and powerful hair dryer works to straighten and smooth delivering beautiful silky hair.",
        "images": "https://s7d9.scene7.com/is/image/BedBathandBeyond/145513347275522p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/98918847339040p?scl=1|https://s7d2.scene7.com/is/image/BedBathandBeyond/10160953308317m?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/10160953308317m?scl=1",
        "name": "Dyson Supersonic Hair Dryer",
        "price": 399.99,
        "url": "https://www.bedbathandbeyond.com/store/product/dyson-supersonic-hair-dryer/3308317",
        "variations": "IRON/FUCHSIA|WHITE/SILVER"
    }
}
,{
    "product": {
        "brand": "KitchenAid",
        "category": "Kitchen|Small Appliances|Mixers & Attachments",
        "currency": "USD",
        "date": "2017-12-07T00:05:25.430Z",
        "description": "This high-performance, 325 watt KitchenAid Artisan Stand Mixer is reason enough for you to get busy in the kitchen. With a 5 qt. ultra durable stainless steel mixing bowl and 10 speed settings, this tilt-back-head all-metal mixer is a kitchen essential.",
        "images": "https://s7d9.scene7.com/is/image/BedBathandBeyond/21686512370920p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/15710817825569p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/68875814073710p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/46977543004843p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/7366314872353p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/18935118698528p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/58050514872485p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685612370938p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/17041218088827p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/31002313317640p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/24925813080976p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/150305412370911p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685714017224p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/5789314222944p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21686413324514p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21686612963238p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685812370962p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/104721943004836p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685912370989p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/109395460419590p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/109395760419613p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21686212863004p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/109395660419606p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685412370903p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/25119914872426p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/58001413227713p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21686012370997p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/7366514872434p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21686112371004p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/31722642049784p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/26824312371012p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/21685312366590p?scl=1|https://s7d1.scene7.com/is/image/BedBathandBeyond/150305412370911p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/150305412370911p?scl=1",
        "name": "KitchenAidВ® ArtisanВ® 5 qt. Stand Mixer",
        "price": 279.99,
        "url": "https://www.bedbathandbeyond.com/store/product/kitchenaid-reg-artisan-reg-5-qt-stand-mixer/102986",
        "variations": "ALMOND|AQUA|BLUE WILLOW|BORDEAUX|BOYSENBERRY|BROWN|BUTTERCUP|COBALT BLUE|CONTOUR SILVER|CRANBERRY|CRYSTAL BLUE|EMPIRE RED|GLOSS CINNAMON|GREEN APPLE|ICE|IMPERIAL BLACK|IMPERIAL GREY|LAVENDER|MAJESTIC YELLOW|MATTE BLACK|MATTE GRAY|METALLIC CHROME|OCEAN DRIVE|ONYX BLACK|PERSIMMON|PINK|PISTACHIO|SILVER|TANGERINE|WATERMELON|WHITE/SILVER|WHITE/WHITE"
    }
}
,{
    "product": {
        "brand": "All-Clad",
        "category": "Gifts|Gifts by Interest|Gifts for the Cook",
        "currency": "USD",
        "date": "2017-12-07T00:05:29.438Z",
        "description": "All-Clad is the first choice of serious cooks. Three-ply bonded construction has a pure aluminum core for even heat distribution and a non-reactive stainless-steel interior and exterior for stick-resistant and easy-to-clean benefits.",
        "images": "https://s7d1.scene7.com/is/image/BedBathandBeyond/1861812460112p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/1861812460112p?scl=1",
        "name": "All-Clad 12-Quart Stainless Steel Multi-Cooker",
        "price": 149.99,
        "sku": "12460112",
        "url": "https://www.bedbathandbeyond.com/store/product/all-clad-12-quart-stainless-steel-multi-cooker/1012460112"
    }
}
,{
    "product": {
        "brand": "Homedics",
        "category": "Health & Beauty|Massage & Relaxation|Massage",
        "currency": "USD",
        "date": "2017-12-07T00:05:30.079Z",
        "description": "Feel the soothing warmth of the HoMedics Shiatsu Neck and Shoulder Massager with the added heat to the shiatsu, vibrating, or combined settings. It's all customizable so you can feel comfortable and natural in your relaxation.",
        "images": "https://s7d1.scene7.com/is/image/BedBathandBeyond/46662342763468p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/46662342763468p?scl=1|https://s7d9.scene7.com/is/image/BedBathandBeyond/46662342763468p__1?scl=1",
        "name": "HoMedicsВ® Shiatsu Neck and Shoulder Massager with Heat",
        "price": 39.99,
        "sku": "42763468",
        "url": "https://www.bedbathandbeyond.com/store/product/homedics-reg-shiatsu-neck-and-shoulder-massager-with-heat/1042763468"
    }
}
,{
    "product": {
        "brand": "Presto",
        "category": "Gifts|Gifts by Category|Unique Gifts",
        "currency": "USD",
        "date": "2017-12-07T00:05:30.730Z",
        "description": "Make delicious, authentic pizza parlor pizza at home. With the exclusive Roto-bake technology you can choose exactly how bubbly the cheese should be and precisely how crispy or chewy you'd like the crust.",
        "images": "https://s7d1.scene7.com/is/image/BedBathandBeyond/397311975038p?scl=1",
        "name": "Presto Pizzazz Pizza Cooker",
        "price": 59.99,
        "sku": "11975038",
        "url": "https://www.bedbathandbeyond.com/store/product/presto-pizzazz-pizza-cooker/1011975038"
    }
}]