# Site grabber

**Site grabber** explores a web site from a page you give it, follows its links a few levels deep, and lists the files it finds: every picture of a gallery, the PDFs of a documentation site, the recordings of a podcast's pages, or the whole site to read offline. You tick the ones you want and DDM downloads them like any other download.

## Grab a site

1. Click **Grabber** on the toolbar, or choose **Tasks → Run site grabber...**.
2. Pick a **Template**: **All images**, **All audio**, **All video**, **All documents**, **Whole site for offline reading**, or **Custom**. A template sets which files are collected and how deep the grabber goes; you can change both later.
3. Type the **Start address**, the page to start from, for example `https://example.com/gallery/`. Without `http://` or `https://`, DDM uses `https://`.
4. If the site asks for a user name and password, type them under **Login**. They are sent to the start address's site only, never to another site its pages link to. A login saved in **Options → Sites Logins** for that site is used too.
5. Click **Next**, choose where the files go (see below), click **Next** again, check what to explore, and click **Start**.

While it works, the window shows how many pages it has explored, how many are waiting, how many files it has found, and the address it is reading. **Stop** ends it at once; what it found so far is kept.

## Where the files go

- **Save to**: a folder of your choice. Left empty, each file goes to its category's folder (**Options → Save to**), as other downloads do.
- **Keep the site's folders**: under the folder, each file goes where it is on the site: `example.com\photos\2025\harbour.jpg`. **Put every file in this one folder** puts them side by side instead. A name that is taken gets a number, as always.
- **Category**: the category the downloads are listed under, or **Each by its file type**.

## What it explores and collects

- **Depth**: how many links away from the start page it reads pages. 0 reads the start page only; 2 reads the pages it links to, and the pages those link to.
- **Explore pages on**: **The same host** (the default; `www.` or not), **The same domain and its subdomains** (`example.com`, `img.example.com`), or **Any site**.
- **Collect files from other sites too**: pictures and files the pages link to on another site (a site's images on a separate image server) are collected too. Untick it to keep to the sites explored.
- **File types**: Images, Video, Audio, Documents, Archives, Programs, **Web pages** (the pages themselves) and **Styles, scripts, fonts** (what pages need to look right). **Other types** takes extensions of your own, separated by spaces: `PSD RAW`.
- **Only files whose address is like** and **Leave out addresses like**: patterns, separated by spaces. `*` stands for any text and `?` for one character; a pattern without them matches any address that contains it. `*/photos/*` collects only files in a `photos` folder; `thumb` leaves out every address with `thumb` in it. Addresses left out are neither explored nor collected. Letter case doesn't count.
- **Most pages** and **Most files**: the grabber stops when it has explored or collected that many, and says so.
- **Pause between requests** and **Requests at once**: how gently it reads the site. The defaults (half a second, two at once) are polite to small sites.
- **Follow the site's robots.txt** (on by default): pages the site asks robots not to visit are not explored, and a `Crawl-delay` it asks for is kept (up to 10 seconds). The window counts the addresses it left out for it.
- **Look up each file's size**: once exploring is done, DDM asks the server each file's size. Untick it for a faster list.

The grabber finds links where a browser does: links, frames, pictures (`srcset` and pictures loaded late too), videos and their posters, sounds, style sheets and the pictures in them, icons. It doesn't run the pages' scripts, so files that only a script adds to a page aren't found. A link whose address doesn't show what it is (`/download?id=7`) is asked once: a page is explored, a file is collected under the name the server gives it.

## Download what it found

When it ends, the window lists the files: their name, type and size. They are all ticked, except those already in your download list.

- Type in **Filter** to show only the files whose name or address contains it. **Select all** and **Select none** tick or untick the files shown.
- **Add to queue** chooses the queue they join, and **Start the queue now** whether it starts. The queue downloads as many at once as it allows.
- **Download selected** adds the ticked files to the download list; **Download all** adds every file found. Each download carries the page it was found on as its referrer, as the browser would.

## Grabber projects

Each grab is kept as a project, listed under **Grabber projects** in the Categories pane. Click a project to see the downloads it added. Right-click it for:

- **Run again**: explore the site again with the same settings (new files on the site are found; files already in the list come unticked).
- **Edit...**: open its settings in Site grabber.
- **Files found...**: the list of files its last run found, to download more of them.
- **Delete**: forget the project. Its downloads stay in the list and its files on the disk.

Projects are saved in the data folder (`grabber.json`). A project's password is saved as the other passwords in the settings are: sealed for your Windows account, and readable only by you on Linux and macOS.

## Next steps

- [Link grabber and archives →](https://dannex.kavangi.co.ke/help/link-grabber.html)
- [Queues and the Scheduler →](https://dannex.kavangi.co.ke/help/queues-and-scheduler.html)
- [Options → Sites Logins →](https://dannex.kavangi.co.ke/help/options.html)
