Workflow

The table is on the page. It just isn't in your sheet

IMPORTHTML breaks the moment a site changes its markup, and a real scraper is a project with maintenance attached. Give SheetAI a URL and describe the columns you want. It fetches the page, reads it, and writes the rows, with the source and the date recorded, so you know exactly what you are looking at.

Try it on your sheet See real workflows

Audit-ready trailNo data trainingReversible per action

51%

Of web traffic is automated

Imperva's 2025 Bad Bot Report found that automated traffic made up 51% of all web traffic in 2024, overtaking human activity for the first time in a decade.

Source: Imperva: 2025 Bad Bot Report

48%

Of top news sites block OpenAI's crawlers

The Reuters Institute found that by the end of 2023, 48% of the most used news websites across ten countries blocked OpenAI's crawlers and 24% blocked Google's AI crawler.

Source: Reuters Institute: How many news websites block AI crawlers? (2024)

28%+

Of key web sources restricted in a year

The Data Provenance Initiative audited 14,000 web domains and found that between 2023 and 2024, new robots.txt rules fully restricted 28% or more of the most actively maintained, critical sources in the C4 dataset.

Source: Data Provenance Initiative: Consent in Crisis (2024)

Web Data to Sheets: illustration for SheetAI's web data to sheets use case

Publicly accessible data is generally scrapeable, but the terms attached to it are not decoration.

In hiQ Labs v. LinkedIn the Ninth Circuit held that scraping data a site makes publicly available is unlikely to violate the US Computer Fraud and Abuse Act. hiQ still lost, on breach of LinkedIn's user agreement. Access and permission are different questions, and only one of them is settled.

Source: hiQ Labs, Inc. v. LinkedIn Corp.: Ninth Circuit

This page is about getting published web data into a spreadsheet: competitor pricing, published rate cards, regulator listings, directory pages, documentation tables, statistical releases, and the endless supply of information that exists as a web page rather than a download. SheetAI fetches the URL, parses what it finds, and writes the rows into the range you name, recording the source and the retrieval date next to them. It reads publicly available pages, it respects what a site asks automated clients not to fetch, and it will not log in as you. The boundaries section is unusually specific here, because this is the workflow where the legal and ethical lines are real.

The state of getting data off the web

Four facts that shape what is realistic in 2026, technically and legally.

Popular sites adding AI-crawler restrictions to robots.txt

Sharply rising

Following the launch of AI-specific user agents, a large share of high-traffic publishers added explicit disallow rules. Any responsible fetching tool has to honour them, which narrows what is legitimately available.

Robots.txt studies, 2024–2025

CFAA exposure for scraping genuinely public pages

Low

hiQ v. LinkedIn and the Supreme Court's narrowing of the CFAA in Van Buren v. United States both point the same way: accessing public data is not 'unauthorised access'. This says nothing about contract terms.

hiQ (9th Cir.) / Van Buren (2021)

Contract and terms-of-service exposure

Real

hiQ ultimately lost on breach of LinkedIn's user agreement. Terms of service, database rights, and copyright in compiled content are where actual liability sits, not the fetch itself.

hiQ v. LinkedIn, final judgment

Selector-based scrapers surviving a site redesign

Rarely

IMPORTHTML, XPath, and CSS-selector extraction all encode assumptions about markup. Redesigns break them silently, producing empty or shifted columns rather than an error.

Practitioner experience

The pattern: the hard part of web data was never the fetch. It is knowing whether you are allowed to, keeping the extraction working when the page changes, and being able to say where a number came from six months later. Those are the three things worth automating.

Anatomy of getting one page into a sheet

What it takes today to turn a published page into rows, done properly.

Phase 1: Check you are allowed

›Read the site's terms of service
›Check robots.txt for the paths you want
›Establish whether the content is licensed or public

⚠ Everyone skips this, and it is the only step with legal consequences. It is also the step that takes four minutes.

Phase 2: Get the page

›Fetch the URL
›Deal with pagination, if there is any
›Handle content rendered after page load

⚠ Half the useful tables in the world are paginated twenty rows at a time, and a meaningful share only exist after JavaScript runs.

Phase 3: Find the data in the page

›Identify which table is the one you want
›Work out the column meanings
›Discard navigation, ads, and boilerplate

⚠ IMPORTHTML makes you count tables by index. Table 3 becomes table 4 after a redesign, and nothing tells you.

Phase 4: Clean what came out

›Strip currency symbols and footnote markers
›Convert text to numbers and dates
›Split merged header rows

⚠ Web tables are formatted for reading, not for arithmetic. Everything arrives as text with decoration attached.

Phase 5: Record where it came from

›Note the URL and the retrieval date
›Keep a copy of what the page said
›Re-run next month and compare

⚠ Without a timestamp, a scraped number is undated and therefore unciteable, and the page will have changed by the time anyone asks.

Cautionary tale

Public does not mean permitted

hiQ Labs built its entire business on data from public LinkedIn profiles. When LinkedIn sent a cease-and-desist, hiQ sued and won a preliminary injunction; the Ninth Circuit held in 2019 and again in 2022 that scraping publicly available data likely falls outside the Computer Fraud and Abuse Act, a position reinforced by the Supreme Court's narrow reading of the CFAA in Van Buren v. United States. hiQ won that argument and lost the case. In late 2022 the district court found hiQ had breached LinkedIn's user agreement, and the parties settled with hiQ under a permanent injunction. The company had already shut down.

The lesson: The lesson is not 'do not fetch web pages'; it is that the fetch is the easy question. Terms of service, database rights, copyright in compiled listings, and personal-data rules are where the exposure actually lives, and none of them are resolved by the page being publicly reachable. SheetAI reads publicly available pages and honours robots.txt directives, and it will not authenticate as you to reach content behind a login. What the site's terms permit you to do with what you retrieved remains your call to make.

Source: hiQ Labs, Inc. v. LinkedIn Corp.: case history

Who it's for

If part of your job is copying published numbers into a spreadsheet, this is the page.

Pricing & Category Managers

Track competitor list prices across dozens of published pages, weekly, by hand.

Market Researchers

Compile industry figures published as web pages that nobody offers as a download.

Policy & Public Affairs

Monitor regulator registers and published decisions that update without notice.

Procurement

Compare published rate cards and framework pricing across suppliers.

Analysts & Journalists

Turn a published table into a dataset that can be cited with a date attached.

Real web-data workflows

The prompt, what the agent fetches, and the table you end up with.

web-scraping_workbook.xlsx - SheetAI

You ask

“Fetch this URL and find the table of published figures. Write it into Data!A1 with the original headers, convert the numbers to numeric values, keep footnote markers in a separate column, and add the source URL and today's date in the last two columns.”

SheetAI does

✓Fetches the page and locates the relevant table by its content rather than its index.
✓Writes headers and rows, stripping formatting decoration from the values.
✓Separates footnote markers so the numeric column stays summable.
✓Records the source URL and retrieval date beside the data.

Result

A 340-row table in the sheet with clean numeric columns, footnotes preserved but out of the way, and provenance attached, citeable rather than merely present.

Everything you need to turn a page into rows

Content-based extraction that survives redesigns, provenance recorded beside every row, and explicit gaps where a page simply does not say.

✓Fetch and parse any publicly reachable URL

✓Content-based table finding, not index-based

✓HTML tables, CSV endpoints, and readable text

✓Source URL and retrieval date per row

✓Snapshot comparison for change monitoring

✓robots.txt directives respected

Plays well with your stack

Public HTML pages and published tables
CSV and JSON endpoints
Government and regulator open-data pages
Documentation and reference sites
Published price and rate-card pages
PDF documents reachable by URL

What a weekly competitor price check looks like

A representative pricing analyst tracking 24 competitor products across four sites every week, before and after the fetch moves into the sheet.

Before SheetAI

~6 hours/week

Mon
Open 24 product pages; copy prices into the tracker
~2.5h
Tue
Chase the pages that changed layout; re-find the prices
~1h
Wed
Compare against last week by eye; build the movement list
~1.5h
Thu
Write the summary; re-check the two that looked wrong
~1h

✓

With SheetAI

~45 minutes/week

Mon
One prompt fetches all 24 with timestamps attached
~15m
Tue
Review the NOT PUBLISHED rows and the outliers
~15m
Wed
Movement list computed against last week's snapshot
~15m

~87% reduction on a representative 24-product weekly price check.

What SheetAI will not do

This is the workflow with the most real-world limits, so they are worth stating plainly rather than burying.

Log in, or work around a paywall or access control

It fetches publicly reachable pages. It does not authenticate as you, replay your session, or circumvent access restrictions. If content sits behind a subscription you pay for, the terms of that subscription govern what you may extract, and that is a conversation with the publisher, not a technical problem to route around.

Ignore robots.txt

If a site's robots.txt asks automated clients not to fetch a path, that request is honoured. This occasionally means a page you can open in a browser is not one this tool will retrieve. That is the intended behaviour, not a limitation to be worked around.

Give you legal clearance

Terms of service, database rights, copyright in compiled listings, and personal-data rules all apply independently of whether a page is publicly reachable, as hiQ Labs discovered after winning the access argument and losing the contract one. Whether you may use what you retrieve, and for what, is a decision for you and your counsel.

Harvest personal data at scale

Compiling personal information about identifiable people from public profiles is a regulated activity under GDPR and comparable regimes regardless of how the data was obtained. SheetAI is built for published commercial, statistical, and reference data, not for building profiles of individuals.

“

We had an IMPORTHTML formula that broke about every six weeks, and it broke silently; you would only notice because a column had gone empty. Now we fetch by describing what we want, and when a site redesigns it still finds the prices. The timestamp column turned out to matter more than any of it; we can finally date our own analysis.

Pricing Analyst, consumer goods distributor (24 SKUs tracked weekly)

Frequently asked

The questions people ask before pointing this at a website.

Is web scraping legal?

›

The honest answer is: it depends on which question you are asking. In the US, hiQ v. LinkedIn and the Supreme Court's narrowing of the CFAA in Van Buren indicate that accessing genuinely public pages is unlikely to be 'unauthorised access' under that statute. But hiQ still lost, on breach of LinkedIn's user agreement. Terms of service, database rights, copyright in compiled content, and data protection rules apply separately. Read the terms of the sites you intend to fetch, and take advice for anything commercially significant.

How is this different from IMPORTHTML or IMPORTXML?

›

Those functions address a table by its position or an element by its path, which means they encode assumptions about a page's markup. When the site is redesigned, the formula does not error; it quietly returns the wrong table or nothing at all. SheetAI reads the page's content and finds the data you described, so a layout change is usually survivable. It also handles pages those functions cannot, and records where each row came from.

Can it handle pages that need JavaScript to render?

›

Partially. Server-rendered content and standard HTML tables are reliable. Content that only appears after client-side rendering is inconsistent, and where it fails you will see missing data rather than a silent wrong answer. For a heavily JavaScript-driven site that you need to track continuously, a purpose-built scraper with a headless browser is the correct tool.

Does it respect robots.txt?

›

Yes. If a site's robots.txt asks automated clients not to fetch a path, the fetch does not happen. This occasionally surprises people: a page opens fine in your browser but will not retrieve here, and that is deliberate. Honouring the directive is the behaviour that keeps this a legitimate tool rather than an evasion one.

Can it scrape behind a login?

›

No. It does not authenticate as you, use your cookies, or replay a session. If the data is behind a login you legitimately hold, check whether the service offers an API or an export; that path is both easier and unambiguously permitted, which is rarely true of the alternative.

How do I track changes over time?

›

Fetch into a dated snapshot sheet and compare against the previous one on a stable identifier. The comparison workflow above does exactly this and returns three lists: added, removed, and changed with the specific fields. Keeping the snapshots is what makes the history defensible rather than anecdotal.

How many URLs can it do at once?

›

Dozens in a pass is routine. Each URL is a real network fetch and a real read, so this is slower than a formula and pleasantly faster than a person. For hundreds of URLs on a repeating schedule, you are describing a pipeline rather than a spreadsheet task.

What if the page has no table, just text?

›

That is fine and often the more useful case. Describe the fields you want and it will extract them from prose: published rates mentioned in a paragraph, a status stated in a sentence, dates buried in a notice. Extraction from text is where selector-based tools have nothing to offer at all.

Sources & further reading

Every benchmark and statistic on this page is drawn from publicly available research. We cite our sources because we read theirs.

Get started

Stop copying from browser tabs.
Start fetching into rows.

Paste a URL, describe the columns you want, and get a dated table in your sheet. No scraper to build, no selectors to maintain.

All use cases

Pay as you go, no subscription required