Парсер товаров для магазина American Eagle Outfitters

American Eagle Outfitters — американский ретейлер одежды и аксессуаров со штаб квартирой в Питтсбурге, Пенсильвания. Бренд был основан в 1977 году братьями Джеем и Марком Сильверман. Используя этот бесплатный парсер, вы сможете собрать информацию о товарах, представленных на сайте ae.com.

Примерное количество товаров: 10000
Примерное количество запросов: 20000
Рекомендуемый план подписки: X-Small

ВНИМАНИЕ! Количество запросов может превышать количество товаров, потому что данные о вариациях, изображениях и др. могут парсится используя запросы к дополнительным ресурсам. Также часть данных о товаре может доставляться с помощью XHR запросов, что также увеличивает общее количество необходимых запросов.

Для его использования вы должны иметь учетную запись в нашем сервисе Diggernaut.

1. Пройдите по этой ссылке для регистрации в сервисе Diggernaut
2. После регистрации и подтверждения email адреса войдите в свою учетную запись
3. Создайте проект с любый именем и описанием, если вы не знаете как, обратитесь к нашей документации
4. Войдите во вновь созданный проект и создайте в нем диггер с любым именем, если вы не знаете как, обратитесь к нашей документации
5. Скопируйте в буфер обмена приведенный ниже сценарий диггера и вставьте его в созданный вами диггер, если вы не знаете как, обратитесь к нашей документации
6. Переключите режим работы диггера с Debug на Active, если вы не знаете как, обратитесь к нашей документации
7. Запустите ваш диггер и дождитесь окончания его работы, если вы не знаете как, обратитесь к нашей документации
8. Скачайте собранный набор данных в нужном вам формате, если вы не знаете как, обратитесь к нашей документации

В дальнейшем вы можете установить расписание для запуска вашего парсера и забирать информацию регулярно.

Сценарий парсера:

---
config:
    debug: 2
do:
- pool_clear: pages
- walk:
    to: https://www.ae.com
    do:
    - find:
        path: a.third-level,a.second-level-item
        do:
        - parse:
            attr: href
        - normalize:
            routine: url
        - link_add:
            pool: catalog
- walk:
    to: links
    pool: catalog
    do:
    - find:
        path: div.product-tile>div.product-details-container>a[data-qa-link-to="pdp"]
        do:
        - parse:
            attr: href
            filter: ^([^\?]+)
        - normalize:
            routine: url
        - link_add:
            pool: pages
- walk:
    to: links
    pool: pages
    do:
    - sleep: 3
    - find:
        path: div.container-fluid[itemtype="http://schema.org/Product"]
        do:
        - variable_clear: pid
        - object_new: product
        - eval:
            routine: js
            body: '(function (){var d = new Date(); return d.toISOString()})();'
        - object_field_set:
            object: product
            field: date
        - static_get: url
        - object_field_set:
            object: product
            field: url
        - find:
            path: h1.psp-product-name
            do:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: name
        - find:
            path: span.pdp-about-cs
            do:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: sku
        - find:
            in: doc
            path: script:contains("var $AWP_ENV")
            do:
            - parse:
                filter: var\s+\$AWP_ENV\s+\=\s+(.+)\s*$
            - normalize:
                routine: json2xml
            - to_block
            - find:
                path: body_safe>viewdata>availableproducts
                do:
                - find:
                    path: brandname
                    slice: 0
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - object_field_set:
                        object: product
                        field: brand
                - find:
                    path: colorname
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - if:
                        match: \w+
                        do:
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: variations
                - find:
                    path: largeimages
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - if:
                        match: \w+
                        do:
                        - normalize:
                            routine: url
                        - variable_set: iurl
                        - register_set: "?scl=1"
                        - object_field_set:
                            object: product
                            joinby: "|"
                            field: images
        - find:
            path: span.psp-product-regularprice[itemprop="priceCurrency"]
            do:
            - parse:
                attr: content
            - object_field_set:
                object: product
                field: currency
        - find:
            path: span.psp-product-regularprice[itemprop="price"]
            do:
            - parse:
                attr: content
            - object_field_set:
                object: product
                type: float
                field: price
        - find:
            path: span.psp-product-saleprice[itemprop="priceCurrency"],span.psp-product-sale-currency[itemprop="priceCurrency"]
            do:
            - parse:
                attr: content
            - object_field_set:
                object: product
                field: currency
        - find:
            path: span.psp-product-saleprice[itemprop="price"]
            do:
            - parse:
                attr: content
            - object_field_set:
                object: product
                type: float
                field: price
        - find:
            in: doc
            path: div.pdp-about-details-equit
            slice: 0
            do:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: description
        - find:
            in: doc
            path: ol.breadcrumb>li.bc-item
            slice: 0:-2
            do:
            - variable_clear: list_item
            - parse
            - space_dedupe
            - trim
            - if:
                match: \w{2,}
                do:
                - object_field_set:
                    object: product
                    joinby: "|"
                    field: category
        - object_save:
            name: product

Ниже приведен пример датасета с несколькими товарами в формате JSON (для наглядности). Датасет может быть скачан и как CSV, XLSX, XML, и любой другой текстовый формат используя темплейтный подход.

[{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|Best Sellers",
        "currency": "USD",
        "date": "2017-12-05T09:15:17.642Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/0341_7976_020_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_020_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_020_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_020_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_020_d1?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_410_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_410_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_410_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_410_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_410_d1?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_639_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_639_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_639_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_639_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_951_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_951_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_951_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_951_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_7976_951_d1?scl=1",
        "name": "AE Chenille Balloon-Sleeve Sweater",
        "price": 26.97,
        "url": "https://www.ae.com/women-gifts-ae-chenille-balloon-sleeve-sweater-light-gray/web/s-prod/0341_7976_951",
        "variations": "Gray|Navy|Rose|Light Gray"
    }
}
,{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|Best Sellers",
        "currency": "USD",
        "date": "2017-12-05T09:15:37.475Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/0341_8032_401_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_401_os?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_401_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_401_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_401_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_401_d1?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_os?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_b?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_500_d1?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_639_of?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_639_os?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_639_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_639_f?scl=1|https://s7d2.scene7.com/is/image/aeo/0341_8032_639_b?scl=1",
        "name": "AE Soft & Pretty Sweater",
        "price": 24.99,
        "url": "https://www.ae.com/women-gifts-ae-soft-pretty-sweater-rose/web/s-prod/0341_8032_639",
        "variations": "Light Blue|Purple|Rose"
    }
}
,{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|New Arrivals|Tops|Sweaters",
        "currency": "USD",
        "date": "2017-12-05T09:15:58.845Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/1341_7951_020_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_020_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_020_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_020_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_401_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_401_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_401_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_401_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_410_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_410_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_410_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_410_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_580_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_580_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_580_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_613_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_613_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_613_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7951_613_b?scl=1",
        "name": "AE Waffle Knit Turtleneck Tunic",
        "price": 32.97,
        "url": "https://www.ae.com/women-gifts-ae-waffle-knit-turtleneck-tunic-gray/web/s-prod/1341_7951_020",
        "variations": "Gray|Light Blue|Navy|Lavender|Burgundy"
    }
}
,{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|New Arrivals|Tops|Sweaters",
        "currency": "USD",
        "date": "2017-12-05T09:16:41.537Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/1341_7947_286_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7947_286_os?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7947_286_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7947_286_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7947_286_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7947_286_d1?scl=1",
        "name": "AE Destroyed Cutout-Collar Sweater",
        "price": 26.97,
        "url": "https://www.ae.com/women-gifts-ae-destroyed-cutout-collar-sweater-oatmeal/web/s-prod/1341_7947_286",
        "variations": "Oatmeal"
    }
}
,{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|Statement Sleeves",
        "currency": "USD",
        "date": "2017-12-05T09:17:09.343Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/1341_7819_008_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_008_os?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_008_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_008_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_008_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_106_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_106_os?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_106_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_106_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_106_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_309_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_309_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_628_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_628_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_628_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_628_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_639_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_639_os?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_639_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_639_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7819_639_b?scl=1",
        "name": "AE Zip-Back Bell Sleeve Sweater",
        "price": 26.97,
        "url": "https://www.ae.com/women-ae-zip-back-bell-sleeve-sweater-neon-pink/web/s-prod/1341_7819_628",
        "variations": "Charcoal|Cream|Olive|Neon Pink|Rose"
    }
}
,{
    "product": {
        "brand": "AEO",
        "category": "Women|Featured|New Arrivals|Tops|Sweaters",
        "currency": "USD",
        "date": "2017-12-05T09:17:29.820Z",
        "description": "Sweater weather forever.",
        "images": "https://s7d2.scene7.com/is/image/aeo/1341_7808_001_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_001_b?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_300_of?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_300_os?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_300_ob?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_300_f?scl=1|https://s7d2.scene7.com/is/image/aeo/1341_7808_300_b?scl=1",
        "name": "AE Chenille Side-Tie Sweater",
        "price": 26.97,
        "url": "https://www.ae.com/women-ae-chenille-side-tie-sweater-green/web/s-prod/1341_7808_300",
        "variations": "Black|Green"
    }
}
]