ask a bot to read web pages into a table

for a small collection of pages, give a bot the urls and the exact fields you want back. it reads them in its own browser and returns a table you can check against the originals. ask it to leave missing values blank and record pages it could not open. this is a supervised reading task, with no promise of complete coverage, bulk crawling or access through a blocked page.

by the dusk bot team · checked 12 november 2026 · how we made this

pixel art: a coral pixel bot, a cyan pixel bot and a mint cat bot beside a desktop computer and a chart on an easel, on a lilac hill in front of sand dunes by the sea, one saying "one row per page"

the routine

send a short list of permitted public urls first. describe each field as you would to someone collecting it by hand. the example task below collects workshop listings, not personal contact details.

paste this into a chat with your bot

every wednesday at 11, read the public workshop pages in the url list i gave you, using your own browser. do not follow links beyond that list.

return one row per listing with: workshop title, organiser, displayed date, displayed location, displayed price and currency, source url, and date read. copy dates and prices as shown; do not calculate missing ones. use “not listed” for a missing field. if a page contains several sessions, keep them in separate rows.

report pages that failed separately. stop if a page blocks access, asks for a login or says automated access is not permitted. do not submit forms, accept terms, collect attendee details or bypass restrictions. keep the result in this chat.

when: wednesdays at 11:00 for the same approved list of pages
needs: a bot with its own browser, permitted urls and a field list; no connector needed

what comes back looks like this (an example, not a real inbox)

example extraction for sam’s fictional workshop directory

workshop | organiser | date shown | location shown | price shown | source
repair basics | priya’s studio | 18 november | north room | £25 | supplied page “repair basics”
wood finishing | jo’s workshop | not listed | east hall | not listed | supplied page “wood finishing”

read: 12 november 2026
failed: “weekend sessions” showed an access restriction. no rows collected from it.
check: the repair basics page also mentions materials; no separate materials price is listed.

define the rows before the run

a page can contain a headline price, an optional extra and an old offer. tell the bot which belongs in the table. keep the displayed wording when it matters: “from £25” should not become a guaranteed £25 booking.

start with pages whose structure you can inspect. compare the first table against those pages yourself before adding more urls. a blank field is useful evidence that the page did not provide an answer. filling it from a guess would make the table harder to trust.

use the browser without promising a crawler

a bot can have a browser separate from your own. Screen lets you watch it work, and Take control lets you step in.¹ that gives you a way to inspect a difficult page, not a guarantee that every page can be read.

it reads pages one by one like a person would; it isn't a bulk crawler. if your real question is what the pages mean, use market research. if you only want to hear about changed pages, use competitor monitoring.

check permission before collecting

respect each site’s terms and robots rules, and the personal data laws that apply to your work. review those before approving a page list; do not treat a publicly visible page as permission for every use. this is a practical boundary, not legal advice.

keep collection to fields needed for the task. for contact lists, the lead research page explains a narrower starting point. a browser task can also fail when content needs a sign-in or never finishes loading. ask for that failure in the result rather than an empty table that looks complete.

the guard rails

approve the page list after checking site terms and robots rules
exclude personal details unless you have established a lawful reason to collect them
keep the source url and reading date with every collected row
treat missing fields and blocked pages as gaps, never as instructions to bypass access

questions

can i scrape websites with ai without writing code?

you can describe a small browser reading task in chat and specify the table you want back. it works best on a short list of pages; check the first table against the originals.

what if the website blocks the bot?

the routine tells it to stop and list the blocked page. review whether another permitted source meets the need. do not ask it to evade the restriction.

can i use the table for lead generation?

start by checking the permitted use and keeping only relevant company information. collection and outreach raise separate questions; getting a row into a table does not settle either one.

sources

drafted with ai help from the dusk bot docs and the sources listed on this page, then checked against them. if something here is wrong or out of date, write to [email protected].