thisago's blog

Fetching books: Being a lazy pirate, arrh

Table of Contents

Anna's Archive is a good library, it has a paid membership, but its gratis fetch is still somewhat convenient.

However it takes some time when you're dealing with a larger amount:

  1. Search .epub books
  2. Enter the book page (/md5/)
  3. Enter the /slow_download/ page
  4. Wait (if applicable) and copy the link to wget.
  5. Convert the .epub to .org with a helper script.

This web flow is inefficient: Mouse clocks, redirects, downloads management…

I want to pay the membership, so I can have access to a stable JSON APIs, but… It is much cooler to just use a little of grep.

So I made some bash scripts :b

Fetching the Book

First, the most important fetcher: You give the anti-bot walled fetching page and your WAF cookies. It gives you the file via wget: 

wgetannasarchive --help

* NAME
wgetannasarchive - Fetches the books from Annas Archive

* SYNOPSIS
wgetannasarchive <cookies-txt> <slow-download-url ...>

* DESCRIPTION
Scrappes the URL from bot-walled page using cookies and run ~wget~. Waits the queue if needed.
** Supported Formats
- =<HOST>/slow_download/<MD5>/0/<MIRROR_NUMBER>=
- =<HOST>/md5/<MD5>=

Converting .epub to .org

To read a .epub in plain text, I convert it to .org. And since epubs are mere HTML pages compressed in a ZIP, a simple helper does the job: 

epub2org --help

* NAME
epub2org - Unpacks EPUB, recurse its HTML files and packs into a single Org file.

* SYNOPSIS
epub2org <epub-file> <out-org-file> [--no-autocleanup]

Searching the website and fetching results

Then, as last instance of convenience, this script:

  1. Does a search in Anna's Archive.
  2. Greps the results URL.
  3. And calls wgetannasarchive for each result.
annasarchivesearchdownload --help

* NAME
annasarchivesearchdownload - Search Anna's Archive and fetch results

* SYNOPSIS
annasarchivesearchdownload <cookies-txt> <per-page-fetch-count> <search-term ...>

* DESCRIPTION
Multi-page iteration supported. Each page contains 50 items by default.

Final flow

Easy as below: 

annasarchivesearchdownload my-cookies.txt ~/Downloads 10 'book from john doe'
epub2orgall ~/Downloads/*.epub

See sources in https://codeberg.org/thisago/smallBashScripts

Hope you have a peaceful sea, dear pirate.