Scraper - v2.0.2

Scraper

github npm

version monthly downloads

Test semantic-release

This tool uses its own parsing of the HTML into JSON, then this package scrolls through it to find the desired information.

Read this for more information. Or visit this website for the API types

Copyright © 2024 DisQada

This tool is licensed under the Apache 2.0 License.
See the LICENSE file for more information.

Getting Started

The package breaks down the HTML into JSON nodes, following the concept that an HTML tag is represented as a JSON object/node.

Each node represents an HTML tag with it's tag name, attributes and children nodes. The children nodes may contain text as string values

Note that the parser completely ignores HTML comments (<!--example-->)

First and most importantly, we need to have our HTML string ready in a variable.

If you're going to use the same HTML string for multiple uses, it's better to parse it yourself and then pass the JSON output to the functions (so the HTML string will be parsed only once). The following example shows how to do this:

import { parse } from '@disqada/scraper'

// Use another package to fetch the HTML from the web
const html = `
<html>
<head>
<title>Test Page</title>
</head>
<body>
<h1 id="title">Hello, world!</h1>
<p class="content">This is a test paragraph.</p>
</body>
</html>
`
const nodes = parse(html)

// Rest of the code ...
import { findNode } from '@disqada/scraper'

const node = findNode(nodes, {
tag: 'h1',
attr: {
key: 'id',
value: 'title'
}
})

// node = {
// type: 'element',
// tagName: 'h1'
// attributes: [{key: 'id', value: 'title' }],
// children: []
// }
import { grabText } from '@disqada/scraper'

const text = grabText(nodes, {
tag: 'p'
})

// text = 'This is a test paragraph.'

The function grabText can be given a TextOptions object that specifies some configurations for the search process. Note that it's optional.

Click on the blue highlighted TextOptions to read more.

import { grabAttr } from '@disqada/scraper'

const attr = grabAttr(nodes, { tag: 'p' }, 'class')

// attr = 'content'

You can download an HTML file and its parsed JSON file under the scrap folder in the root path of your project outside runtime by calling the download command CLI.

Note that either --url or --path must be given

Arg name required Description
--file true Name of the downloaded HTML and parsed JSON file
--url false Link of the web page
--path false Path of a local HTML file (the HTML file will be copied to the scrap folder)
scraper --url='https://example.com/sample' --file='sample'
scraper --path='./samples/v1/index.html' --file='sample1'