How is the Internet Archive using BitTorrent?
As of summer 2012, the Internet Archive is beta-testing the distribution of our public collections via the BitTorrent protocol (as a supplement to traditional HTTP download).
Currently over 1.4 million Archive Items are available via the BitTorrent protocol, comprising almost a petabyte of public domain materials.
As testing continues, more and more content will be made available through Torrents. For the details, see the related FAQ, Details of Archive-made Torrents.
BitTorrent download requires an up-to-date BitTorrent client.
For general information on the BitTorrent protocol, see Wikipedia or BitTorrent.com.
How do I find Torrents on the Archive?
You can search and browse all our Torrents on the Torrents collection homepage (or one of the media-specific subcollections).
To narrow your own Search or Advanced Search query, add format:”Archive BitTorrent” to your search terms, e.g. https://archive.org/search.php?query=’scifi AND mediatype:audio AND format:”Archive BitTorrent”‘.
The most popular and recent Torrents are available on each tracker’s hotlists, e.g. bt1.archive.org Hot List.
How do I edit metadata of my item?
For existing items use this clickpath from the item’s details page:
Edit > change the information > modify/add metadata as desired > click the “Submit” button at the bottom of the page.
You can only modify items that you created.
Can I download only part of an item using an Archive BitTorrent?
Yes, almost all contemporary BitTorrent clients allow you to select which files included in the Torrent are downloaded. And even when you download only one or some files, you get the speed advantages of using the format.
Many show a list of the files contained in the Torrent, and both folders and individual files can be selected or deselected both before, and during, download.
It is recommend, in fact, that you deselect the top-level directory within the Torrent named ._____padding_file if there is one, as this contains unnecessary (empty) Internet Archive padding files.
My Torrent download never completes?
Most likely, you have an out-of-date Torrent for the Item you are trying to download. The first thing to try is re-downloading the Items’ Torrent, and trying again.
Torrents for Items on the Internet Archive can become obsolete when the Item the Torrent is for changes. In that case, some or (more rarely) all of the files within the Torrent will fail to download completely.
This is because our Torrents rely heavily on webseeding (download directly from our servers, when no peers have the files you are seeking). When files on our servers have changed since the Torrent was made, they will not match expected ‘piece hashes’; some BitTorrent clients (e.g. Transmission) will attempt to re-download file pieces from changed files over and over, forever, assuming there was an error in transmission, when in fact the file has changed.
Torrents that never download at all most likely are the result of a different problem, lack of client support for Getright-style webseeding.
My Torrent download never starts?
It’s worth mentioning that some BitTorrent clients take a very long time to begin downloading when relying on webseeding (a common requirement when using Archive BitTorrents). At times downloads can take upwards of several minutes to start.
We’re not sure exactly why; we suspect those clients exhaust all other options, such as DHT, before falling back on webseeds. (We have observed this behavior with Transmission.)
If you download an up-to-date (current) Torrent from the Archive, and it loads into your BitTorrent client, but download never begins, the most likely cause is that you are using a BitTorrent client that does not support Getright-style webseeding.
Our Torrents rely heavily on webseeding (download directly from our servers, when no peers have the files you are seeking). Some BitTorrent clients (e.g. rTorrent) do not support Getright-style webseeding, and will not be able to download un-seeded Internet Archive Torrents.
At the moment, the only solution to this problem is to use a different client.
Another possibility is that your Torrent file is out of date, because the Item has moved to a new server, and your client does not support redirection of our canonical webseeding URL (and no tracked or discoverable peers are seeding the Torrent).
In this case, the problem can be solved by re-downloading the Torrent file.
How do I tell if a Torrent is being seeded?
Current seed and leech counts are displayed for each Archive Torrent on the relevant Item details pages, in parenthesis next to the Torrent link. These values are cached for five minutes or so, and because clients do not always update our trackers regularly, they may be somewhat out of date.
The number of seeders is shown first, and the number of leechers (downloaders without the complete Torrent) second. The seeder number includes ‘webseeds,’ however, which are only usable by BitTorrent clients that support Getright-style webseeding.
Does the Internet Archive run trackers?
Yes, Internet Archive torrents are tracked by bt1.archive.org and bt2.archive.org.
We are using opentracker, which has proven to be highly scalable.
Our trackers are closed (they track our only own torrents).
How do I use Torrents to upload to archive.org?
Retrieval of Torrents is not the best solution for uploading unless you already have an existing mechanism for creating and seeding Torrents.
This capability is not intended as an alternative to our uploader. It merely enables the Archive to capture material already being distributed via BitTorrent.
Torrent retrieval by the Archive works like this:
If a valid .torrent file is uploaded (e.g. through our Uploader) into an item, when that item is derived, we will instantiate a BitTorrent client (Transmission) and attempt to retrieve the Torrent. If the Torrent is successfully retrieved, its contents will be added to the item. ‘Valid’ in this case means, well-formed and seeded.
Our client will attempt to scrape any listed trackers to find seeding peers, but will also attempt to find peers via DHT and can fall back on Getright-style webseeding when possible.
The Torrent file itself is leeched only long enough to retrieve the file; we do not seed the Torrent after retrieval.
However, all items contents, including those retrieved through this method, are made available via the item’s own Archive Torrent. (Because it contains additional contents, this Archive Torrent will, alas, have a different infohash from the original Torrent. So uploading a Torrent to the Archive does not make us a seeder of it.)
Bonus feature: if you have only a magnet link, and not a Torrent file, you can create a dummy .torrent file by pasting that magnet link into a text file and naming it foo.torrent.
If you upload this dummy Torrent file, we’ll detect that you gave us a magnet link and take care of the rest.
Uploading BitTorrents to the Internet Archive
Starting in 2011, the Internet Archive began automatically retrieving BitTorrent files uploaded into most Community collections.
Uploading a Torrent provides a convenient way to upload many files or large contents, provided seeds (including webseeds) are available for the Torrent.
How to prevent an Archive Torrent from being made
Internet Archive BitTorrents are automatically made for community-contributed items in many collections, and automatically updated when item contents or metadata change.
If you prefer that your item not have an Archive Torrent made for it; or that items within a collection you maintain do not, you can prevent Torrents from being made by including the following metadata tag in your item:
noarchivetorrent=true
Note: adding this tag does not remove existing Torrents, those must be removed using the Item Manager item file management tool.
For instructions on how to edit an item or collection’s metadata, see the FAQ Uploading Content.
Why is the Torrent link for an Item lined out (Torrent)?
While an Item is being updated, its Torrent link is temporarily disabled and shown as Torrent.
Changes to an item usually render any existing Archive BitTorrent for it obsolete. Attempts to download obsolete Archive Torrents will usually fail, as described here: My Torrent download never completes?. (Technically, the problem is that when files within an Item change, they can no longer download correctly via webseeding because the piece hashes for updated files change).
The Torrent link will return to normal when the Item finishes updating and the torrent is updated. The Torrent link may be unavailable for a few minutes or a few hours depending on the size of the Item and how busy the Archive processing cluster is (in very rare cases, it might be disabled for a day or more).
Note: obsolete torrents will continue to be tracked by Archive trackers for some time, but will only be retrievable when seeded by peers who have downloaded the referenced version of the item.
What are peers, seeds, leechers, and snatches?
BitTorrent is a peer-to-peer file-sharing protocol facilitated by centralized trackers. The Internet Archive runs several BitTorrent trackers to allow for peer discovery.
Archive trackers track (but do not log or otherwise record) which peers have pieces of which Torrents; real-time statistics are summarized on tracker hotlists for each of our Trackers.
Internet Archive tracker statistics of interest
Peers: the total number of clients known by the tracker to have pieces of a Torrent, i.e. the sum of seeds and leechers.
Seeds: the number of clients known by the tracker to have all of the pieces of a Torrent available, i.e. those which have downloaded the entire Torrent but remain online.
Leechers: the number of clients known by the tracker to have some of the pieces of a Torrent available, i.e. those currently downloading the Torrent.
Snatches: the number of clients known by the tracker to have downloaded a given Torrent, but which are not currently seeding it.
Note: Internet Archive seeder and peer counts include webseeds; these seeds are available only when using clients that support Getright-style webseeding.
My item does not have a torrent. How can I add one?
You would need to send a request to have a torrent generated to info@archive.org. Please include the URL of the item page.