Setting up Calibre and FanFicFare for Downloading from AO3

First a note about how I read fanfic. I prefer to read on my kindle as I find it less distracting and easier on my eyes than my phone or pc. So some formatting in here is just for getting the files ready to be put into Send to Kindle for cloud access. You can alter some of the end steps for side-loading to a kindle or kobo, or into Books on an Apple device. It just depends on what device you wish to use to read.

I archive with epub because it's currently the most widely acccepted format. It also works with Send to Kindle and converts decently well. It does not however support a wide variety of emojis, and can have issues with creative fonts. If you want to side-load onto a kindle you will need to convert to AZW3, KFX (may require additional plugins) or the very old mobi format. Mobi may be required if you have an extremely old kindle otherwise AZW3 is the current standard.

Calibre if first and foremost a database. It will automatically setup folders for each author and manage where your files are saved. It does not play well with google drive or other "live" backup systems. I have it running fine with a backup drive that runs daily but I know it doesn't like anything that tries to mess with it's file structure.

You will need a pc or laptop for this and I use a Windows machine so the instructions will be geared towards that ecosystem. It doesn't need to be a particularly new machine, but Windows 10 or 11 is what these instructions will be using. You can install older versions of calibre on older machines and most of it will work similarly. The main difference will be inability to download a series as a single epub.

Shortcuts

Setting up Calibre | Setting Up FanFicFare | Other Useful Calibre Plugins | Downloading Fic | Creating Custom (Text) Covers | Save to Disk Conversion Settings | Using Send to Kindle and Collections | Custom Dictionaries on Kindle

Setting Up Calibre

First you need to Download Calibre and run the basic install. Pick the version for your machine, I use the 64-bit for Windows. Currently, I'm at version 7.8 though there may be newer ones by now. Install calibre with whatever standard settings come up.

Custom Columns

Think about what info you want to keep connected to your downloads. Also consider what data you want to display in the list view. I have several hundred older fics I downloaded with other means and they now lack useful data I keep from new downloads (they also largely lost italics or bold as well...txt files are not good for archiving but I can open them unlike the old wps files.)

I keep major plot points (tags), pairings, word count, fandom and series at a minimum.

Now that you've decided what info you want to keep make some custom columns. Just right click on any column header and choose Add Your Own Columns from the dropdown list.

In the bottom left click + Add custom column and you get this popup:

You only need to fill in the first three items but can fill it all in if you want.

And here are the options for Column Type, I only use a few of them.

Annoyingly, Calibre will want to restart after each column addition or edit. My custom columns are as follows:

Decide what info you want to keep from fics and create some columns to hold it. You can add more later, but it gets annoying if you've already downloaded a large number of fics to update. I have 3,000 something and when I recently added Word Count it was a mess.

Existing Backups

Now might be a good time to import any older files you've got for fanfiction. I know not everyone has hundreds of older files, but I imported all the old mobi and epub files I downloaded with FanFicDownloader and added what info I had to my new columns. If the file is from AO3 FanFicFare can actually extract some info and possibly download newer versions of fic as well. I'll detail how it does that at the end of the Downloading Fic section.

Calibre can convert several formats to epub well and some others badly. I mostly converted .doc and .txt files, and had to go in and create table of contents after. PDF - > epub conversion is particularly awful, though correctly formatted text may work fine.

Setting Up FanFicFare

Documentation: GitHub, and MobileReads

Easiest install is within Calibre's plugin section. In the upper menu click on Preferences. Scroll down to Advanced -> Plugins. Inside Plugins just Search for FanFicFare. Install it with a button in one of the menus, the default is the main toolbar.

Once installed you may need to restart calibre, but then it's time to configure it. Click the dropdown to the right of the FanFicFare icon and choose Configure FanFicFare. Second from the bottom on the version I'm using.

You'll get a popup with lots and lots of options and several tabs. Only some of it will be important. Everything has tooltips if you hover and it's mostly self explanatry. See below for the settings I use on the tabs I use.

Basics Tab

Under Updating Calibre Metadata I have Search inside ebooks for Story URL checked as I had a large backlog of old fics downloaded with a different downloader and wanted to update to new metadata without overwriting the fics. The above setting lets FanFicFare add the Story URL to the fic metadata, then it could use it to update the metadata.

Personal.ini

This is where you put your AO3 login info and add a line to let it download "adult" files without nagging. I think you may not needs adult true set if you have your login included for AO3 anyway, but I set this up so long ago I have no idea now. Here's the two things I've added to personal.ini. You would add your own username and password.

Standard Columns

Where you pick what info gets updated when. I have tags added only for a new fic so I can have incomplete fics in my database without losing my custom tags. I don't care to keep all tags so this is fine for me.

Custom Columns

Where you hook your custom columns up to the FanFicFare standard info.

Other Useful Calibre Plugins

These install the same as FanFicFare, just go to the plugins section of Preferences and search for them. You will probably want a button added to the menu somewhere for both when you set them up.

EpubMerge

This plugin is required if you want to download a series as an anthology file with FanFicFare. It can also be used to merge older fics you may have already saved. It will add a table of contents to the new grouped epub.

Generate Cover

This plugin lets you dynamically populate a custom cover with info from you custom columns. My custom covers include: Series (if applicable), Title, Word Count (added later so missing on some of my older downloads), Tags, Pairings and Author. Calibre has a standard cover generator as well, but I like how this one handles dynamic data.

Downloading Fic

So, all your columns are set up and ready to go. Time for a few test fics to make sure everything is going into the correct columns. I'd pick some with lots of tags and multiple chapters just to make completely sure it works as intended.

Single Fic

Click the arrow on the right of the FanFicFare button, select Download from URL and Paste the URLS. You can paste more than one at a time, just select in the box and hit Enter or Return for a new line and paste the next URL.

Select what you want to do from the dropdown.

Series Merges into Anthology

Click the arrow on right of FanFicFare button, select Get Story URLS from Webpage and Paste the page URL. Select For Anthology Epub. Note: for an Anthology you are limited to 20 URLS into one epub, or at least to how many ever URLS show on a page. So if a series has multiple pages you have to do each individually as AO3 only displays 20 per page.

There will be a second popup where it lists the URLS. Be aware it will find ANY URLS on the page, so if any fic description includes a link to a podfic version or something that will be included.

I edit the Titles of these removing Anthology and adding [1-X] with X being the total number of fics so I know how many are in the file. I also add a new tag labeled Series so I know it was an Anthology.

Page with List of Individual Fics

Click the arrow on right of FanFicFare button, select Get Story URLS from Webpage and Paste the page URL. Select For Individual Books. This will download from a search, filtered tag page or from the main works page from an author.

Updating Previously Downloaded Fic

I have a few incomplete fic I've downloaded with FanFicFare. As a new chapter is added I tend to update them. If a fic is incomplete I give it an "INCOM" tag so I can filter just for them. To update a fic with new chapters you just select the fic and hit the FanFicFare button. Dpending on what you pick in the Update Mode dropdown you can download just a new chapter or redownload the whole fic replacing the existing one, such as if the author says they made edits to previous chapters. I tend to just download the new chapters. Select what you want to do from the dropdown.

All of these will have similar popups that show up while they're working, one for MetaData in the center of the screen while it's running and one will eventually appear in the bottom right you have to click for it to actually make the epubs. It will also have a log if it couldn't download for some reason.

Updating Older Fanfics

So you have super old mobi or text files you saved 15 years ago? FanFicFare has a setting to scan through a file and fine a story URL (mostly works for AO3). It can then use the story url to update the fic. I mostly did this with old mobi files. Convert the file to epub, and just click the FanFicFare button. Since I set "Search ebooks for Story URL?" to yes when I set up FanFicFare it will scan the file and assign the URL. I mostly used this to pull tags and wordcount, but you can also use it to force update the story depending on what you pick on the popup the FanFicFare creates. You just change the Update Mode.

Fanfiction.net Downloading Headache

For fics that are on Fanfiction.net I use FicHub as FanFicFare doesn't download them easily. There are some workarounds that can be used, but I haven't bothered looking them up when FicHub works. I just import into Calibre and fix any tags manually. For me this is for re-downloading older fics I saved badly as text files as I don't read on fanfiction.net anymore.

Tags and Pairings Cleanup

I prune tags viciously before I archive a fic. I have a column that's hidden keeping all of them, so I tend to cut out anything incidental for the ones I'll use on a cover. I also created a shorthand for me to identify plot points in a fic from what was put on the cover. What you read will dictate what tags you want to include or delete.

I cut Tags down to a shortend Fandom name, and plot points. Some fandoms get further segemented by character, plot or timeline. For some example tropes Time Travel becomes TT, Alternate Universe becomes AU, I keep ones like ABO, Soulmate or Crack, because they will help me filter what I may want to read at a given time. Anything I have a lot or very few of I keep to segement fics without keeping every tag on a cover.

I also used to include a tag for length before I started pulling word count. I don't really need them anymore, but I keep using them because for some reason Ficlet is easier to read than the word count if I'm scrolling.

I truncate pairings as well. Mostly down to first name / first name only keeping the main pairings. Since everythign is set to a fandom this works for me. Some background ones I'll include if they're particularly pertinent. If the fic is Gen I'll often include just the main characters as pairings tags. I started this more recently as I have two or three fandoms where everything is Gen and it's getting messy finding anything.

What you want to keep will depend on what you read. The shorter you make them the more you can store on a cover and still read it as a thumbnail.

Editing Fics?

The main time I do this is removing extra blank paragraphs. AO3 adds a lot of space between paragraphs anyway, but some authors add another empty paragraph between due to how AO3 converts from whatever they copied it from. It makes reading extremely annoying on a kindle as I get double the "pages" to turn. I go into Edit Book, and do a Find/Replace for the empty P tag copied from one of the xhtml files then just save.

I once went in and fixed the spelling format of a name I couldn't stand reading they way they wrote it. Another time a fic was set up as emails between characters and FanFicFare ate the email addresses for "privacy" so made it completely unredable.

If you have a super old badly formatted fic you may need to dig a bit to separate it into chapters or add a Table of Contents. This is not going be covered here. You'd do better searching for "how to edit a book in calibre" on your search engine of choice. It's basically just html and css though in another package.

Creating Custom (Text) Covers

Calibre has options for background images and colors for the covers. I don't use that as I read 99% of the time on an e-ink kindle so a white background with black text is most visible for me. If I was using a color tablet I might segment background color by fandom or something. There are also some fics that have custom covers that will download with FanFicFare. Some of those I keep if I like them.

After I've cleaned up my tags I hit Generate Cover (which you will only have if you downloaded the "Generate Cover" plugin. I have several templates based on the length of the metadata I'm including. Some have smaller or larger fonts depending on title length or just overall content. A long series name with a long title can eat a lot of page space.

You will want to research what size you want your covers. I use 1600 x 2560, which I found on some Amazon listing for kindle book sizes a while ago. It will show black bars on different edges depending on device, but it works for me.

I'm not creative with these, I was shooting for readability when on a tiny phone screen if I was reading on my kindle app, I only recently upgraded from an iPhone SE. Fonts are Arial Black for Title and Author and just Arial for everything else, 150pt. Margins 200 top and bottom, 40 on the sides, 30 for text spacing (basically the line-height).

Here's an example of my standard cover settings:

Image even though it's unchecked needs to be where it is in this list or it breaks and puts the author on the top I have no real idea why... but probably related to margin settings.

My Custom Text:
{#length}
{tags:sublist(0,0,\,)}
{#pairings:sublist(0,0,\,)}

length is the Lookup Name for my Word Count column. tags is just the normal tags column. pairings, is the lookup name for my Pairings column. The # symbols just label them as custom columns.

My Series text: {series_index} of {series}

Save to Disk Conversion Settings

I use Save to disk for all my Send to Kindle files. Using Save to disk lets me make sure the epub covers are getting embedded and the filenames are set up so sorting on kindle works the way I want. This is under Preferences -> Saving books to disk.

This is my Save Template: {series} {series_index:0>2s|| - } {title:re(:, -)}

If a file has a series it will be put first in the filename, followed by a dash and the series number with an initial 0 if it is less than 2 digits. If there's no series it will just be the title of the book with any ":" replaced with a "-". Calibre auto replaces ":" with an underscore, and I just prefer a dash.

I keep an empty folder on my desktop to save my files into. They then get dragged into Send to Kindle.

Using Send to Kindle and Collections

This step is infinitely easier on a tablet app but it works on pc and is painfully slow on an e-ink kindle. There are ways to get calibre to email files direct to Send to Kindle. I don't use it that way so I can't describe how to do it.

Creating Collections

If you have an Amazon account you have a Content library. If you share an amazon account just note that anything you Send to Kindle may end up shared. You can send to just an individual kindle app or device, but you won't have cloud functionality. The Content Library has Collections for digital content, mostly books, but it will also work for "Documents" which is what uploaded fanfic will be considered.

I have 57 collections at the moment, most are just an overall fandom. Some are a smaller chunk of a fandom such as "SW - Pre-Empire" and "SW - Mando" for Star Wars vs just one giant collection. I try to keep the number of fics per collection to a few hundred at most. Anything above that and finding specific things gets messy. I also have standalone collections, such as "Books - Read" and "Rereads - Long". Rereads being my favorite fics I like to reread occasionally.

Decide how you want to segment to collections and just Click "Add Collection". They will filter out to all your kindle devices though it may take a few minutes. Once you have the collections you want you can start adding fics.

I use the Send to Kindle page on Amazon though there are also desktop apps and you can upload direct from a phone. Collections can also be created on any device and it will filter over to the other devices logged into your account.

I upload in batches by collection, waiting for each batch to show in the Documents view then mass selecting them and adding to a Collection.

If you already have your fics uploaded and are now adding collections this is where a tablet is easier. You can select multiple files easily and add them to a collection. It's possibly on an e-ink kindle but I've found it much more annoying to do as you can only select more than one in a limited screen, and doing it one at a time is torturous.

Custom Dictionaries on Kindle

This is only tangentally related to the above tutorial. I've been reading Star Wars fanfic, specially Mandalorian Star Wars pieces. There are lots of authors using Mandalorian words, or making up their own based on a partial language from the official novels. Most add a definitions section at the end of each chapter, or use in-line popups. The popups don't translate over to epubs on my kindle. My solution was looking up custom dictionaries for kindle. They're possible though a little annoying to impliment. But once they're on there they function the same as the official dictionaries with the popup when you highlight a word.

I used a combination of these two sites for definitions:

I also lightly scrolled a few Wiki sites for planet names and species descritions for aliens.

I used mostly these two sites to figure out how to format the dictionary.

Then I spent entirely too long formatting words to include various tenses so the dictionary "worked". Example of a single entry for a verb.

<idx:entry name="default" scriptable="yes" spell="yes">
<h4><dt><idx:orth>emuurir
<idx:infl inflgrp="verb">
<idx:iform name="infinitive" value="emuurir" />
<idx:iform name="conjugated" value="emuuri" />
<idx:iform name="past" value="r'emuuri" />
<idx:iform name="future" value="ven emuuri" />
<idx:iform name="imperative" value="k'emuuri" />
</idx:infl>
</idx:orth></dt>
</h4><dd>[eh-MOO-reer]<br/>to enjoy, to like</dd>
</idx:entry>
<hr/>

Other than calibre I needed an html editor, I use Notepad++ and the Kindle Previewer application to package it together. I sideloaded it into my Kindle Paperwhite Dictionaries folder then set it to the default dictionary.

It doesn't work quite as well as I'd like as it overrides the default dictionary so if I look up a normal English word it now defaults to Wikipedia and the "normal" dictionary is in the third popup under a dropdown, but I don't need to learn Mando'a either, so a win for me.