Парсер информации с сайта магазина Calypso St Barth о товарах и ценах

С момента запуска своего первого магазина в 1992, компания Calypso St. Barth выросла из скромного бутика курортной одежды до роскошного, узнаваемого во всем мире бренда. Дизайнеры, вдохновленные путешествиями, культурой и исскуством разных стран, создают удивительные и женственные модели, которые нравятся женщинам разных возрастов. Если вам по душе смесь легкой элегантности и экзотических идей, вам обязательно понравится ассортимент этого интернет-магазина. Парсер информации с сайта магазина Calypso St Barth о товарах и ценах позволит вам извлечь также изображения, если это необходимо для решения вашей задачи.

Примерное количество товаров: 2000
Примерное количество запросов: 2000
Рекомендуемый план подписки: Free

ВНИМАНИЕ! Количество запросов может превышать количество товаров, потому что данные о вариациях, изображениях и др. могут парсится используя запросы к дополнительным ресурсам. Также часть данных о товаре может доставляться с помощью XHR запросов, что также увеличивает общее количество необходимых запросов.

Как пользоваться парсером информации о товарах и ценах

Для использования парсера информации с сайта магазина вы должны иметь учетную запись в нашем сервисе Diggernaut.

1. Пройдите по этой ссылке для регистрации в сервисе Diggernaut
2. После регистрации и подтверждения email адреса войдите в свою учетную запись
3. Создайте проект с любый именем и описанием, если вы не знаете как, обратитесь к нашей документации
4. Войдите во вновь созданный проект и создайте в нем диггер с любым именем, если вы не знаете как, обратитесь к нашей документации
5. Скопируйте в буфер обмена приведенный ниже сценарий диггера и вставьте его в созданный вами диггер, если вы не знаете как, обратитесь к нашей документации
6. Переключите режим работы диггера с Debug на Active, если вы не знаете как, обратитесь к нашей документации
7. Запустите ваш диггер и дождитесь окончания его работы, если вы не знаете как, обратитесь к нашей документации
8. Скачайте собранный набор данных в нужном вам формате, если вы не знаете как, обратитесь к нашей документации

В дальнейшем вы можете установить расписание для запуска вашего парсера и забирать информацию регулярно.

Сценарий парсера

---
config:
    debug: 2
    agent: Firefox
do:
- walk:
    to: http://www.calypsostbarth.com/new-arrivals/clothing
    do:
    - find: 
        path: .col-side > ul > li 
        do: 
        - variable_clear: cat1
        - variable_clear: cat2
        - variable_clear: cat3
        - find: 
            path: a
            slice: 0
            do: 
            - parse
            - space_dedupe
            - trim
            - normalize:
                routine: lower
            - variable_set: cat1
        - parse:
            format: html
        - register_set: 

        - to_block
        - find: 
            path: div > ul > li > ul > li 
            do: 
            - variable_clear: cat2
            - variable_clear: cat3
            - find: 
                path: a
                slice: 0
                do: 
                - parse
                - space_dedupe
                - trim
                - normalize:
                    routine: lower
                - variable_set: cat2
                - parse:
                    attr: href
                - normalize:
                    routine: url
                - pool_clear: default
                - parse:
                    attr: href
                - link_add
                - walk:
                    to: links
                    do:
                    - find: 
                        path: .paging  li > a:matches([Nn]ext) 
                        do: 
                        - parse:
                            attr: href
                        - link_add
                    - find: 
                        path: .item > a 
                        do: 
                        - parse:
                            attr: href
                        - walk:
                            to: value
                            do:
                            - object_new: product
                            - find: 
                                in: doc
                                path: head 
                                do: 
                                - eval:
                                    routine: js
                                    body: '(function (){var d = new Date(); return d.toISOString()})();'
                                - object_field_set:
                                    object: product
                                    field: date
                                - static_get: url
                                - object_field_set:
                                    object: product
                                    field: url
                            - find: 
                                path: .tabbed li:nth-of-type(1) 
                                do: 
                                - parse
                                - space_dedupe
                                - trim
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: description
                            - find: 
                                path: body 
                                do: 
                                - register_set: Calypso St. Barth
                                - object_field_set:
                                    object: product
                                    field: brand
                                - variable_get: cat1
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                                - variable_get: cat2
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                                - variable_get: cat3
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                            - find: 
                                path: .product-name > h1 
                                do: 
                                - parse
                                - space_dedupe
                                - trim
                                - object_field_set:
                                    object: product
                                    field: name
                            - find: 
                                path: img.cloudzoom-gallery, .image > img
                                do: 
                                - parse:
                                    attr: data-cloudzoom
                                - eval:
                                    routine: js
                                    body: (function () {var e = {}; return JSON.stringify(e);})();
                                - normalize:
                                    routine: json2xml
                                - to_block
                                - find: 
                                    path: zoomImage 
                                    do: 
                                    - parse
                                    - normalize:
                                        routine: url
                                    - object_field_set:
                                        object: product
                                        field: images
                                        joinby: "|"
                            - find: 
                                path: script:matches(var\s*spConfig) 
                                do: 
                                - parse:
                                    filter:
                                        - Product\.Config\((.+)\);
                                - normalize:
                                    routine: json2xml
                                - to_block
                                - find: 
                                    path: 'attributes > *:hasChild(code:matches(color)) > options > label'
                                    do: 
                                    - parse
                                    - space_dedupe
                                    - trim
                                    - object_field_set:
                                        object: product
                                        field: variations
                                        joinby: "|"
                                - find: 
                                    path: body_safe > productid 
                                    do: 
                                    - parse
                                    - space_dedupe
                                    - trim
                                    - object_field_set:
                                        object: product
                                        field: sku
                                - find: 
                                    path: body_safe > baseprice
                                    do: 
                                    - parse
                                    - if:
                                        match: (\d)
                                        do:
                                        - object_field_set:
                                            object: product
                                            field: price
                                            type: float
                                        - register_set: USD
                                        - object_field_set:
                                            object: product
                                            field: currency
                            - object_save:
                                name: product
            - find: 
                path: ul > li > a 
                do: 
                - parse
                - space_dedupe
                - trim
                - normalize:
                    routine: lower
                - variable_set: cat3
                - parse:
                    attr: href
                - normalize:
                    routine: url
                - pool_clear: default
                - parse:
                    attr: href
                - link_add
                - walk:
                    to: links
                    do:
                    - find: 
                        path: .paging  li > a:matches([Nn]ext) 
                        do: 
                        - parse:
                            attr: href
                        - link_add
                    - find: 
                        path: .item > a 
                        do: 
                        - parse:
                            attr: href
                        - walk:
                            to: value
                            do:
                            - object_new: product
                            - find: 
                                in: doc
                                path: head 
                                do: 
                                - eval:
                                    routine: js
                                    body: '(function (){var d = new Date(); return d.toISOString()})();'
                                - object_field_set:
                                    object: product
                                    field: date
                                - static_get: url
                                - object_field_set:
                                    object: product
                                    field: url
                            - find: 
                                path: .tabbed li:nth-of-type(1) 
                                do: 
                                - parse
                                - space_dedupe
                                - trim
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: description
                            - find: 
                                path: body 
                                do: 
                                - register_set: Calypso St. Barth
                                - object_field_set:
                                    object: product
                                    field: brand
                                - variable_get: cat1
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                                - variable_get: cat2
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                                - variable_get: cat3
                                - if:
                                    match: (\S)
                                    do:
                                    - object_field_set:
                                        object: product
                                        field: categories
                                        joinby: "|"
                            - find: 
                                path: .product-name > h1 
                                do: 
                                - parse
                                - space_dedupe
                                - trim
                                - object_field_set:
                                    object: product
                                    field: name
                                
                            - find: 
                                path: img.cloudzoom-gallery, .image > img
                                do: 
                                - parse:
                                    attr: data-cloudzoom
                                - eval:
                                    routine: js
                                    body: (function () {var e = {}; return JSON.stringify(e);})();
                                - normalize:
                                    routine: json2xml
                                - to_block
                                - find: 
                                    path: zoomImage 
                                    do: 
                                    - parse
                                    - normalize:
                                        routine: url
                                    - object_field_set:
                                        object: product
                                        field: images
                                        joinby: "|"
                            - find: 
                                path: script:matches(var\s*spConfig) 
                                do: 
                                - parse:
                                    filter:
                                        - Product\.Config\((.+)\);
                                - normalize:
                                    routine: json2xml
                                - to_block
                                - find: 
                                    path: 'attributes > *:hasChild(code:matches(color)) > options > label'
                                    do: 
                                    - parse
                                    - space_dedupe
                                    - trim
                                    - object_field_set:
                                        object: product
                                        field: variations
                                        joinby: "|"
                                - find: 
                                    path: body_safe > productid 
                                    do: 
                                    - parse
                                    - space_dedupe
                                    - trim
                                    - object_field_set:
                                        object: product
                                        field: sku
                                - find: 
                                    path: body_safe > baseprice 
                                    do: 
                                    - parse
                                    - if:
                                        match: (\d)
                                        do:
                                        - object_field_set:
                                            object: product
                                            field: price
                                            type: float
                                        - register_set: USD
                                        - object_field_set:
                                            object: product
                                            field: currency
                            - object_save:
                                name: product

Пример данных, собранных парсером с сайта

Ниже приведен пример датасета с несколькими товарами в формате JSON (для наглядности). Датасет может быть скачан и как CSV, XLSX, XML, и любой другой текстовый формат используя темплейтный подход.

[{
    "product": {
        "brand": "Calypso St. Barth",
        "categories": "new arrivals|clothing",
        "currency": "USD",
        "date": "2017-12-27T09:48:47.804Z",
        "description": "By CALYPSO St. Barth - Welcome cooler weather with this lavish suede jacket. Supple soft and chic, this jacket flatters with a feminine fit and lengthy silhouette. Accented with a ruffled trim, this autumn favorite features hook and eye closure and welt pockets. Wear with anything from casual basics to night out ensembles, a special wardrobe addition!",
        "images": "http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket4_1.1502486097.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket3_1.1502486097.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket2_1.1502486098.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket5_1.1502486098.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket_1.1502486099.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket.1502486099.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/d/i/diosajacket4_1.1502486097.jpg",
        "name": "Diosa Suede Ruffle Jacket",
        "price": 1100,
        "sku": "86060",
        "url": "http://www.calypsostbarth.com/new-arrivals/clothing/diosa-suede-ruffle-jacket",
        "variations": "ATMN GLD"
    }
}
,{
    "product": {
        "brand": "Calypso St. Barth",
        "categories": "new arrivals|clothing",
        "currency": "USD",
        "date": "2017-12-27T09:48:48.837Z",
        "description": "By CALYPSO St. Barth - Transcend into a new dimension of the open cardigan and welcome a versatile piece, essential to both dressy & casual outfitting. Composed of cozy alpaca wool, this sweater stands in for a lavish cold weather scarf, poncho or transitional jacket. Accented with ribbed detailing at the opening flaps, the cardigan’s lengthy back center seam connects to inside out seam at waist for figure enhancing structure amidst an oversized silhouette. Wear it over dresses, pants or loungewear – it’s the perfect piece to carry you into the colder seasons.",
        "images": "http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artatlaskosencocoon_20.1504818057.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artatlaskosencocoon_35.1504818057.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artatlaskosencocoon_12_ppn_.1504818058.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artatlaskosencocoon_28_ppn_.1504818059.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/s/e/serpinadress_11_1.1504818059.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artatlaskosencocoon_20.1504818057.jpg",
        "name": "Kosen Wool Sweater Cocoon",
        "price": 395,
        "sku": "86562",
        "url": "http://www.calypsostbarth.com/new-arrivals/clothing/kosen-wool-sweater-cocoon",
        "variations": "LT GREY"
    }
}
,{
    "product": {
        "brand": "Calypso St. Barth",
        "categories": "new arrivals|clothing",
        "currency": "USD",
        "date": "2017-12-27T09:48:49.120Z",
        "description": "By CALYPSO St. Barth - Casual dressing made easy, this lightweight cotton dress is comfortable, casual and oh so chic. A relaxed silhouette, this effortless mini is graced with ruffled flounce sleeves. Back yoke and square neckline are accented with shirring details that flow into a loose body. Finished with side seam pockets, this dress will your new “go-to” favorite this season.",
        "images": "http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadress4.1503599057.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadress2.1503599058.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadress3.1503599058.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadressmain.1503599059.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadress.1503599060.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/e/m/emptyname_13.1503599060.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/a/r/artemiadress4.1503599057.jpg",
        "name": "Artemia Flounce Sleeve Dress",
        "price": 225,
        "sku": "86442",
        "url": "http://www.calypsostbarth.com/new-arrivals/clothing/artemia-flounce-sleeve-dress",
        "variations": "Black"
    }
}
,{
    "product": {
        "brand": "Calypso St. Barth",
        "categories": "new arrivals|clothing",
        "currency": "USD",
        "date": "2017-12-27T09:48:49.430Z",
        "description": "By CALYPSO St. Barth - Boho chic meets outdoorsy elegance in this fleece poncho. Adorned with a tribal motif in earthy grey hues, it is finished with olive green crochet detailing along the lower trim, front zip and pockets as well as the shoulder sleeves. Gold zippers are enhanced with taupe leather trim for an added touch of bohemian charm. Luxuriously soft and cozy, this is the perfect outerwear piece to take you into the fall season",
        "images": "http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/b/a/bahamutponcho_26.1504799605.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/b/a/bahamutponcho_46.1504799605.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/b/a/bahamutponcho_85.1504799606.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/b/a/bahamutponcho_79.1504799607.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/1/1/11_2_1640127.1504799608.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/w/i/winterlooks39684.1504799608.jpg|http://www.calypsostbarth.com/media/catalog/product/cache/1/image/975x/040ec09b1e35df139433887a97daa66f/b/a/bahamutponcho_26.1504799605.jpg",
        "name": "Bahamut Fleece Pullover Poncho",
        "price": 395,
        "sku": "86563",
        "url": "http://www.calypsostbarth.com/new-arrivals/clothing/bahamut-fleece-pullover-poncho-86639",
        "variations": "Grey CC"
    }
}]