All Versions
23
Latest Version
Avg Release Cycle
21 days
Latest Release
663 days ago

Changelog History
Page 1

  • v3.1.2 Changes

    November 15, 2022

    🐛 Bug Fixes

    • injectJQuery in context does not survive navs (#1661) (493a7cf)
    • 👉 make router error message more helpful for undefined routes (#1678) (ab359d8)
    • MemoryStorage: correctly respect the desc option (#1666) (b5f37f6)
    • requestHandlerTimeout timing (#1660) (493ea0c)
    • 👯 shallow clone browserPoolOptions before normalization (#1665) (22467ca)
    • 👌 support headfull mode in playwright js project template (ea2e61b)
    • 👌 support headfull mode in puppeteer js project template (e6aceb8)

    🔋 Features

  • v3.1.1 Changes

    November 07, 2022

    🐛 Bug Fixes

    🔋 Features

    • ➕ add static set and useStorageClient shortcuts to Configuration (2e66fa2)
    • ✅ enable migration testing (#1583) (ee3a68f)
    • playwright: disable animations when taking screenshots (#1601) (4e63034)

  • v3.1.0 Changes

    October 13, 2022

    🐛 Bug Fixes

    • ➕ add overload for KeyValueStore.getValue with defaultValue (#1541) (e3cb509)
    • ➕ add retry attempts to methods in CLI (#1588) (9142e59)
    • 👍 allow label in enqueueLinksByClickingElements options (#1525) (18b7c25)
    • basic-crawler: handle request.noRetry after errorHandler (#1542) (2a2040e)
    • 🏗 build storage classes by using this instead of the class (#1596) (2b14eb7)
    • correct some typing exports (#1527) (4a136e5)
    • do not hide stack trace of (retried) Type/Syntax/ReferenceErrors (469b4b5)
    • enqueueLinks: ensure the enqueue strategy is respected alongside user patterns (#1509) (2b0eeed)
    • enqueueLinks: prevent useless request creations when filtering by user patterns (#1510) (cb8fe36)
    • 📦 export Cookie from crawlee metapackage (7b02ceb)
    • 🖐 handle redirect cookies (#1521) (2f7fc7c)
    • http-crawler: do not hang on POST without payload (#1546) (8c87390)
    • ✂ remove undeclared dependency on core package from puppeteer utils (827ae60)
    • 👌 support TypeScript 4.8 (#1507) (4c3a504)
    • wait for persist state listeners to run when event manager closes (#1481) (aa550ed)

    🔋 Features

  • v3.0.4 Changes

    August 22, 2022

    🔋 Features

    • ⬆️ bump puppeteer support to 15.1

    🐛 Bug Fixes

    • key value stores emitting an error when multiple write promises ran in parallel (#1460) (f201cca)
    • 🛠 fix dockerfiles in project templates

  • v3.0.3 Changes

    August 11, 2022

    🛠 Fixes

    • ➕ add missing configuration to CheerioCrawler constructor (#1432)
    • sendRequest types (#1445)
    • 💻 respect headless option in browser crawlers (#1455)
    • 👉 make CheerioCrawlerOptions type more loose (d871d8c)
    • 👌 improve dockerfiles and project templates (7c21a64)

    🔋 Features

    • ➕ add utils.playwright.blockRequests() (#1447)
    • http-crawler (#1440)
    • prefer /INPUT.json files for KeyValueStore.getInput() (#1453)
    • jsdom-crawler (#1451)
    • ➕ add RetryRequestError + add error to the context for BC (#1443)
    • ➕ add keepAlive to crawler options (#1452)

  • v3.0.2 Changes

    July 28, 2022

    🛠 Fixes

    • regression in resolving the base url for enqueue link filtering (1422)
    • 👌 improve file saving on memory storage (1421)
    • ➕ add UserData type argument to CheerioCrawlingContext and related interfaces (1424)
    • always limit desiredConcurrency to the value of maxConcurrency (bcb689d)
    • wait for storage to finish before resolving crawler.run() (9d62d56)
    • using explicitly typed router with CheerioCrawler (07b7e69)
    • 📦 declare dependency on ow in @crawlee/cheerio package (be59f99)
    • 👉 use crawlee@^3.0.0 in the CLI templates (6426f22)
    • 🛠 fix building projects with TS when puppeteer and playwright are not installed (1404)
    • enqueueLinks should respect full URL of the current request for relative link resolution (1427)
    • 0️⃣ use desiredConcurrency: 10 as the default for CheerioCrawler (1428)

    🔋 Features

    • 🔧 feat: allow configuring what status codes will cause session retirement (1423)
    • 👍 feat: add support for middlewares to the Router via use method (1431)

  • v3.0.1 Changes

    July 26, 2022

    🛠 Fixes

    • ✂ remove JSONData generic type arg from CheerioCrawler in (#1402)
    • 0️⃣ rename default storage folder to just storage in (#1403)
    • ✂ remove trailing slash for proxyUrl in (#1405)
    • 0️⃣ run browser crawlers in headless mode by default in (#1409)
    • 📇 rename interface FailedRequestHandler to ErrorHandler in (#1410)
    • 0️⃣ ensure default route is not ignored in CheerioCrawler in (#1411)
    • ➕ add headless option to BrowserCrawlerOptions in (#1412)
    • 🖨 processing custom cookies in (#1414)
    • enqueue link not finding relative links if the checked page is redirected in (#1416)
    • 🛠 fix building projects with TS when puppeteer and playwright are not installed in (#1404)
    • 💻 calling enqueueLinks in browser crawler on page without any links in (385ca27)
    • 👌 improve error message when no default route provided in (04c3b6a)

    🔋 Features

    • 📜 feat: add parseWithCheerio for puppeteer & playwright in (#1418)

  • v3.0.0 Changes

    July 13, 2022

    🚀 This section summarizes most of the breaking changes between Crawlee (v3) and Apify SDK (v2). Crawlee is the spiritual successor to Apify SDK, so we decided to keep the versioning and release Crawlee as v3.

    Crawlee vs Apify SDK

    📦 Up until version 3 of apify, the package contained both scraping related tools and Apify platform related helper methods. With v3 we are splitting the whole project into two main parts:

    • 📦 Crawlee, the new web-scraping library, available as crawlee package on NPM
    • 📦 Apify SDK, helpers for the Apify platform, available as apify package on NPM

    📦 Moreover, the Crawlee library is published as several packages under @crawlee namespace:

    • @crawlee/core: the base for all the crawler implementations, also contains things like Request, RequestQueue, RequestList or Dataset classes
    • @crawlee/basic: exports BasicCrawler
    • @crawlee/cheerio: exports CheerioCrawler
    • 💻 @crawlee/browser: exports BrowserCrawler (which is used for creating @crawlee/playwright and @crawlee/puppeteer)
    • @crawlee/playwright: exports PlaywrightCrawler
    • @crawlee/puppeteer: exports PuppeteerCrawler
    • @crawlee/memory-storage: @apify/storage-local alternative
    • 📦 @crawlee/browser-pool: previously browser-pool package
    • @crawlee/utils: utility methods
    • @crawlee/types: holds TS interfaces mainly about the StorageClient
    Installing Crawlee

    🚀 > As Crawlee is not yet released as latest, we need to install from the next distribution tag!

    📦 Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. @crawlee/playwright if you plan on using playwright - it already contains everything from the @crawlee/browser package, which includes everything from @crawlee/basic, which includes everything from @crawlee/core.

    npm install crawlee@next

    👍 Or if all we need is cheerio support, we can install only @crawlee/cheerio

    npm install @crawlee/cheerio@next

    When using playwright or puppeteer, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.

    npm install crawlee@next playwright

  • v2.3.2 Changes

    May 05, 2022
    • 🛠 fix: use default user agent for playwright with chrome instead of the default "headless UA"
    • 🛠 fix: always hide webdriver of chrome browsers

  • v2.3.1 Changes

    May 03, 2022
    • 🛠 fix: utils.apifyClient early instantiation (#1330)
    • feat: utils.playwright.injectJQuery() (#1337)
    • feat: add keyValueStore option to Statistics class (#1345)
    • 🛠 fix: ensure failed req count is correct when using RequestList (#1347)
    • 🛠 fix: random puppeteer crawler (running in headful mode) failure (#1348) > This should help with the We either navigate top level or have old version of the navigated frame bug in puppeteer.
    • 🛠 fix: allow returning falsy values in RequestTransform's return type