How to Use BitTornado With HIP or With IPv6

NOTE: this manual was written with Ubuntu and other debian based distros in mind. Some parts of this manual may not applicable on RH and others, but the basic idea is still quite similar.

This page describes my experiments with Bittorrent protocol and Host Identity Protocol. I will assume that you (reader) have working knowledge of Bittorrent and HIP, so I will not go over details unless they are totally necessary. I will first describe the test network, software, and what changes I had to make to get my tests working. After the descriptions I will walk through the simplest use case. As a conclusion I will list/describe some things that did not find a proper place in the text body. The title says "or with IPv6" because as you might have already noticed HITs are like IPv6 addresses.

Used software

Ubuntu Jaunty Jackalope 9.04
bittornado-0.3.18 (can be obtained by "apt-get source bittornado")
HIP for Linux (can be obtained from hipl.hiit.fi)

Test setup

Seeder - IPv4: 192.168.1.15, IPv6: 3::4, HIT: 2001:16:fb2:59ba:b19b:8679:8f4:2ac6
Tracker - IPv4: 192.168.1.13, IPv6: 3::5, HIT: 2001:14:fb81:8537:6fd7:2f4f:f961:8f92
Downloader - IPv4: 192.168.1.16, IPv6: 3::3 HIT: 2001:1e:d709:1980:5c6a:bb0c:7650:a027

Torrent players

Fixes

Most of the problems in this experiment resulted from IPv6 support of Bittornado. It almost worked but the tracker had problems with clients that did not use compact mode (read "used IPv6"). There is a patch circulating on some of the torrent forums that disables the compact mode on the clients but I found that "--ipv6_enabled 1" on client and "--ipv6_enabled 1 --compact_reqd 0" on tracker has pretty much the same effect. So I did not use that patch. Tracker worked OK for IPv4 but with IPv6 you only got a traceback (below).

Traceback (most recent call last): File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/RawServer.py", line 144, in listen_forever self.sockethandler.handle_events(events) File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events s.handler.data_came_in(s, data) File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/HTTPHandler.py", line 155, in data_came_in if not c.data_came_in(data) and not c.closed: File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/HTTPHandler.py", line 46, in data_came_in self.next_func = self.next_func(val) File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/HTTPHandler.py", line 78, in read_header r = self.handler.getfunc(self, self.path, self.headers) File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/BT1/track.py", line 951, in get return_type, rsize, params('supportcrypto')) File "/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/BT1/track.py", line 782, in peerlist cache = self.cached.setdefault(infohash,[None,None,None])[return_type] IndexError: list index out of range

So I found myself debugging the tracker and found the problem. When using IPv6, the compact_reqd should be off. One place in the code did not take this into consideration and resulted in in-correctly initialized dictionary. Patch below has a simple fix for the problem (track.py is in <path-to>bittornado-0.3.18/BitTornado/BT1/).

--- track.py.orig 2009-07-23 13:51:24.000000000 +0300 +++ track.py 2009-07-23 13:48:51.000000000 +0300 @@ -773,7 +773,10 @@ data['peers'] = [] return data l_get_size = int(float(rsize)*(len_l)/(len_l+len_s)) - cache = self.cached.setdefault(infohash,[None,None,None])[return_type] + if self.config['compact_reqd']: + cache = self.cached.setdefault(infohash,[None,None,None])[return_type] + else: + cache = self.cached.setdefault(infohash,[None,None,None,None,None])[return_type] if cache and ( not cache[1] or (is_seed and len(cache[1]) < rsize) or len(cache[1]) < l_get_size

Using BitTornado with HIP

Now that I got everything working I made test seeds/downloads. In the following the steps are numbered and illustrated in the Fig above. The commands are explained after the command is shown. Note, I was running BitTornado from the its source folder. Note, replace HITs from the examples with your HITs and file names accordingly.

First briefly about the flow of the experiment (see Fig. above). First, the Seeder creates the torrent metafile from the file that will be shared. Second, Seeder will upload it to some torrent forum or otherwise distribute it to the downloaders. Third, Seeder will contact the tracker and register to it. Fourth, the Downloader(s) will get the torrent metafile. Fifth the Downloader contacts the Tracker based on the information on the torrent metafile. Sixth, the Tracker will send a peer list to the Downloader. Seventh, the Downloader contacts the Seeder and the file is downloaded.

Tracker (not in the Fig. but tracker has to be started before its used :) )

"./bttrack.py --port 6969 --ipv6_enabled 1 --compact_reqd 0 --bind 2001:14:fb81:8537:6fd7:2f4f:f961:8f92 --dfile dstate"

If you are running tests inside on LAN you will need to add "--nat_check 0". This will prevent the trackers NAT checks that will result to traceback and render the tracker useless. As a note, this could be left off always with HIP because HIP has its own NAT traversal (see HIPL manual or the Internet draft). Port parameter defines the port that the tracker listens to. Ipv6_enabled turns the IPv6 support on. Turning off compact_reqd will result in correctly initialized memory database. Bind will tell the tracker what address is used. Note, for HIP we use HIT but this could simply be an IPv6 address. Dfile is a file that is used to save the memory database for it to persist over restarts and so on. If file specified does not exist it will be created automatically. Note. in my tests I noticed that removing the file after failed tries was a good idea, otherwise it contained incorrect data. See "man bttrack" for more information on the usage.

Makemetafile (Step 1. and 2. in Fig. above)

"./btmakemetafile.py http://[2001:14:fb81:8537:6fd7:2f4f:f961:8f92]:6969/announce TestFile"

Seeder uses btmakemetafile to make the actual torrent file that is uploaded to a forum or otherwise distributed. The first parameter is the trackers announce URL and the second parameter is the file to be shared. The trackers announce URL is simply an IP-port combination. Note that we are using HIP and the IP is replaced with the trackers HIT. See "man btmakemetafile" for more information on the usage.

Seeder (Step 3. in Fig. above)

"./btdownloadcurses.py --ipv6_enabled 1 --bind 2001:16:fb2:59ba:b19b:8679:8f4:2ac6 TestFile.torrent"

This can be replaced with btdownloadgui or with btdownloadheadless. They all use the same downloader and are just front-ends for it. I used the curses one and occasionally the "gui" one. The parameters are similar to the trackers parameters. Seeder additionally takes the metafile as the last parameter.

Downloader (Steps 5. - 7. in Fig. above and step 4. to get the torrent)

"./btdownloadcurses.py --ipv6_enabled 1 --bind 2001:1e:d709:1980:5c6a:bb0c:7650:a027 TestFile.torrent"

Seeder and Downloader are the same and their role is assigned based on the fact that do they have the shared data or not (the one with the data is the seeder).

Miscellaneous

For HIT resolution I used host files but it can be replaced with dynamic DNS support or "DHT" support.

By writing http://[2001:14:fb81:8537:6fd7:2f4f:f961:8f92]:6969 to the address bar of your browser will get you to the trackers statistics page.

On some Ubuntu machines python-crypto package needs to be installed

Package wxpython or python-wxgtk (don't remember the name now) may cause problems or is not installed. Installing will fix it. Needed by btdownloadgui.

You might get a deprecation warning (below) about sha module. It does not matter now. Eventually it has to be fixed but ...

/home/sklvarjo/projects/bittornado-0.3.18/BitTornado/__init__.py:8: DeprecationWarning: the sha module is deprecated; use the hashlib module instead from sha import sha

btdownloadcurses tracebacked during downloads, which resulted in weird behaviour in the GUI but the download finished OK. Restarting the GUI said that the download succeeded (Files checksum matched the one in the torrent file). If someone is interested the tracebacks are below.

These errors occurred during execution: [20:35:39] Traceback (most recent call last): File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/RawServer.py", line 142, in listen_forever self.sockethandler.handle_events(events) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events s.handler.data_came_in(s, data) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 403, in data_came_in self.read(s) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 488, in _read2 x = self.next_func(m) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 369, in read_message self.connecter.got_message(self, s) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Connecter.py", line 325, in got_message if c.download.got_piece(i, toint(message[5:9]), message[9:]): File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Downloader.py", line 182, in got_piece self._request_more() File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Downloader.py", line 189, in _request_more self.fix_download_endgame(new_unchoke) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Downloader.py", line 256, in fix_download_endgame del want[self.backlog - len(self.active_requests):] TypeError: slice indices must be integers or None or have an __index__ method [20:35:39] Traceback (most recent call last): File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/RawServer.py", line 142, in listen_forever self.sockethandler.handle_events(events) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events s.handler.data_came_in(s, data) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 403, in data_came_in self.read(s) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 493, in _read2 self.close() File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 382, in close self.sever() File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Encrypter.py", line 391, in sever self.connecter.connection_lost(self) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Connecter.py", line 222, in connection_lost c.download.disconnected() File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Downloader.py", line 88, in disconnected self._letgo() File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/Downloader.py", line 101, in _letgo self.downloader.storage.request_lost(index, begin, length) File "/home/sklvarjo/projects/torrent/bittornado/bittornado-0.3.18/BitTornado/BT1/StorageWrapper.py", line 738, in request_lost assert not (begin, length) in self.inactive_requests[index] AssertionError

All of these manuals/tutorials are provided as is. They worked for me and that is all the help I give with them, so if I forgot something or there is a typo you can inform me but do not expect me to solve your problems :) Oh and almost forgot, use them at your own risk.