Skip to content

Commit f538903

Browse files
author
fabien
committed
2024-11-29 update (MP3)
1 parent 0958e72 commit f538903

File tree

1 file changed

+261
-0
lines changed

1 file changed

+261
-0
lines changed

‎MP3.ipynb

+261
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "44bdd8b4",
6+
"metadata": {},
7+
"source": [
8+
"# PROGRES 2024 - Mini-Projet 3\n",
9+
"# DHT CAN (Content-Adressable Network)\n",
10+
"\n",
11+
"Fabien Mathieu - [email protected]\n",
12+
"\n",
13+
"Sébastien Tixeuil - [email protected]"
14+
]
15+
},
16+
{
17+
"cell_type": "markdown",
18+
"id": "f27c5f0d",
19+
"metadata": {},
20+
"source": [
21+
"The goal of this mini-project is to build a DHT simulator and to test its performance:\n",
22+
"Table construction, distributed content management, load balancing and traffic analysis.\n",
23+
"\n",
24+
"The table will be used to manage Wikipedia pages. **Exceptionally**, the peer in charge of a page will also be responsible for retrieving and making available the content of the page. Recall that this is not the typical behavior of a DHT, which is only supposed to point to the peers owning the content. This change is introduced here in order to reduce the implementation complexity of the mini-project."
25+
]
26+
},
27+
{
28+
"cell_type": "markdown",
29+
"id": "38aba0fe",
30+
"metadata": {},
31+
"source": [
32+
"# Rules"
33+
]
34+
},
35+
{
36+
"cell_type": "markdown",
37+
"id": "4720e742",
38+
"metadata": {},
39+
"source": [
40+
"1. Cite your sources\n",
41+
"2. One file to rule them all\n",
42+
"3. Explain\n",
43+
"4. Execute your code\n",
44+
"\n",
45+
"\n",
46+
"https://github.com/balouf/progres/blob/main/rules.ipynb"
47+
]
48+
},
49+
{
50+
"cell_type": "markdown",
51+
"id": "b9df04ae",
52+
"metadata": {},
53+
"source": [
54+
"# Exercise 1. Creation of a Peer class"
55+
]
56+
},
57+
{
58+
"cell_type": "markdown",
59+
"id": "f6a5e41f",
60+
"metadata": {},
61+
"source": [
62+
"Realize in Python a `Peer` class allowing to simulate a DHT CAN as seen in course.\n",
63+
"\n",
64+
"The communication will be simulated through a dictionary `dht`: `id` -> `Peer`: knowing the id of a peer allows to contact it (in real life, in addition to the id the IP address and port must be provided), and the network calls within the DHT will be represented by calls to the id of the destination in the dictionary.\n",
65+
"\n",
66+
"The hash function to use is SHA1, available from the `hashlib` library https://docs.python.org/3/library/hashlib.html\n",
67+
"\n",
68+
"SHA1 provides 160-bit hashes.\n",
69+
"\n",
70+
"To simplify things, we will assume that the CAN space is a *square* of side $2^{80}$. *Square* means that you don't teleport from the top of to the bottom or from left to right as you do in a *torus*.\n",
71+
"\n",
72+
"The hash will be split in two: the first half will represent the `x` coordinate inside the square, the second half the `y` coordinate inside the square."
73+
]
74+
},
75+
{
76+
"cell_type": "markdown",
77+
"id": "d1da7497",
78+
"metadata": {},
79+
"source": [
80+
"Instances of the `Peer` class will have to implement (at least) the following attributes / methods:\n",
81+
"- `id`: peer identifier (integer)\n",
82+
"- `area`: the coordinates of area the peer is responsible of, e.g. a tuple $(x_\\min, x_\\max, y_\\min, y_\\max)$\n",
83+
"- `neighbors` : the list of the peer neighbors. Each neighbors should be represented by a tuple (id, responsibility_area)\n",
84+
"- `is_in_charge(key)`: a function that indicates whether the peer is in charge of `key`\n",
85+
"- `lookup(key, dht)`: a function that returns the id of the peer in charge of `key` as well as the distance, in number of hops, that separates the peer performing the *lookup* from the peer in charge.\n",
86+
"- `replace(old_neighbor, new_neighbor, new_neighbor_area)`: a function that removes a neighbor from `neighbors` and replace it with `new_neighbor` (and its area)\n",
87+
"- `share_area_with(other_peer, dht)`: a function that splits the responsibility in two\n",
88+
" - If current area is a square, split it vertically; otherwise split it horizontally\n",
89+
" - Each of the two peers (this one and the other) updates its neighbor list to keep the ones side-adjacent to it. Remind that the two peers are also side-adjacent to each other. Also note that it is possible for a neighbor to end up in both peers\n",
90+
" - Don't forget to ask the neighbors that need it to replace the current peer with other_peer in their own neighbor lists\n",
91+
" - Split the table between the two peers (cf below for table)\n",
92+
"- `join(dht, entry_point)`: if the dht is empty, the peer should inserts in it with full resposibility area, e.g. $(0, 2^{80}, 0, 2^{80})$. Otherwise, it should ask through the entry_point for the peer responsible of its id and ask that peer to share its area.\n",
93+
"- `table`: see below \n",
94+
"- `publish_page(page_title, dht)`: see below\n",
95+
"- `retrieve_page(page_title, dht)`: see below\n",
96+
"- `search(search_string, dht)`: see below"
97+
]
98+
},
99+
{
100+
"cell_type": "markdown",
101+
"id": "ae3f59eb",
102+
"metadata": {},
103+
"source": [
104+
"The `table` attribute contains the dictionary of keys managed by the peer; the table may contain two types of keys:\n",
105+
" - word keys: to a given word we associate the hash coordinates of the word prefixed by `word:` (e.g. `computer` is represented by the SHA1 hash-coordinates of `word:computer`, i.e. (688843315192981992985613, 1084012856129654940495670)). The associated value in the table is a set (`set` Python), which will typically contain the titles of wikipedia pages that contain the word.\n",
106+
" - page keys: to a given wikipedia page we associate the hash coordinates of the page title prefixed by `page:` (for example `Circuit (computer science)` is represented by the SHA1 hash coordinates of `page:Circuit (computer science)`, i.e. (1042659198402659147098924, 95359921870751321224488)). The associated value in the table is the content of the page."
107+
]
108+
},
109+
{
110+
"cell_type": "markdown",
111+
"id": "9956f32c",
112+
"metadata": {},
113+
"source": [
114+
"The `publish_page` method should perform the following actions:\n",
115+
"- Separate the title into words written in lower case. For example, `Circuit (computer science)` becomes `[\"circuit\", \"computer\", \"science\"]`, `Object-oriented programming` becomes `[\"object\", \"oriented\", \"programming\"]`.\n",
116+
"- For each word, find the peer responsible for the word key. If this key exists, add the page title to the associated set, otherwise initialize the key (with the value of a one-element set, the page title). Remember that the update is done by the responsible peer, not by the peer calling the method.\n",
117+
"\n",
118+
"\n",
119+
"\n",
120+
"\n",
121+
"The `publish_page` method is not in charge of retrieving the content of the page itself."
122+
]
123+
},
124+
{
125+
"cell_type": "markdown",
126+
"id": "809019cd",
127+
"metadata": {},
128+
"source": [
129+
"The `retrieve_page` method is responsible for providing the content. It should perform the following actions:\n",
130+
"- If the peer is responsible for the page, return the content:\n",
131+
" - If the key is present in the table, return the associated value\n",
132+
" - If the key is not present, use the wikipedia module to retrieve the page from the title (method `wikipedia.page`), store the content (attribute `content` of the result) in the table and return it\n",
133+
" - In case of an error from the wikipedia module (this can happen, but quite rarely), store nothing and return ``Page not found``.\n",
134+
"- If the peer is not responsible for the page, retrieve the content from the responsible peer."
135+
]
136+
},
137+
{
138+
"cell_type": "markdown",
139+
"id": "50cc8dcc",
140+
"metadata": {},
141+
"source": [
142+
"The `search` method should: \n",
143+
"- Separate the search into words written in lower case.\n",
144+
"- Retrieve for each word the set of Wikipedia pages containing the word (if the word is not indexed, it is the empty set)\n",
145+
"- Return the list of titles containing all the words of the search.\n",
146+
"- If the list is empty but at least one word is indexed, issue a warning (see https://docs.python.org/3/library/logging.html) and return the union (titles containing at least one of the words of the search).\n",
147+
"- If no word is indexed, issue a warning that none of the words were found and return an empty list."
148+
]
149+
},
150+
{
151+
"cell_type": "markdown",
152+
"id": "b316ea7e",
153+
"metadata": {},
154+
"source": [
155+
"The initialization of a peer will take one argument: its `id`."
156+
]
157+
},
158+
{
159+
"cell_type": "markdown",
160+
"id": "87a099a6",
161+
"metadata": {},
162+
"source": [
163+
"**Note**: `table`, `publish_page`, `retrieve_page` and `search` will only be used from exercise 3 on. You can code a first version that does not contain these attributes/methods, and update at the time of exercise 3."
164+
]
165+
},
166+
{
167+
"cell_type": "markdown",
168+
"id": "bde11615",
169+
"metadata": {},
170+
"source": [
171+
"# Exercise 2. DHT creation"
172+
]
173+
},
174+
{
175+
"cell_type": "markdown",
176+
"id": "abda4dda",
177+
"metadata": {},
178+
"source": [
179+
"From an empty `dht` dictionary, insert 10000 pairs of identifier 0 to 9999 in this order, using 0 as the insertion point.\n",
180+
"- What is the average size of the neighbor list? The size of the smallest list? The size of the bigger list?\n",
181+
"- What is the average logarithm of the area (in base 2)? The logarithm of the smallest area? The logarithm of the largest area?\n",
182+
"- From the peer with identifier 42, identify with the `lookup` method the distance (in hops) that separates it from the successors of each of the 10000 peers. What is the average distance? The greatest distance?\n",
183+
"- Are the results in line with what you saw in the course? Explain your statement."
184+
]
185+
},
186+
{
187+
"cell_type": "markdown",
188+
"id": "e162d542",
189+
"metadata": {},
190+
"source": [
191+
"# Exercise 3. Publishing pages from Wikipedia"
192+
]
193+
},
194+
{
195+
"cell_type": "markdown",
196+
"id": "1a8d948e",
197+
"metadata": {},
198+
"source": [
199+
"Using the wikipedia module and the `links` method of the `WikipediaPage` objects, retrieve the set of pages located at most 2 hops away from the `Computer science` page, i.e. :\n",
200+
"- `Computer science`\n",
201+
"- The titles of the pages referenced by the `Computer science` page\n",
202+
"- The titles of the pages referenced by the pages referenced by the `Computer science` page\n",
203+
"- After eliminating duplicates, save the list of titles to a file (so you can re-use it later) and publish all the titles in the DHT. You can use an arbitrary peer to call the method.\n",
204+
"\n",
205+
"- How many words are indexed in total?\n",
206+
"- What is the average size of the peer tables? The size of the largest table?"
207+
]
208+
},
209+
{
210+
"cell_type": "markdown",
211+
"id": "c0a039f9",
212+
"metadata": {},
213+
"source": [
214+
"# Exercise 4. Using the DHT"
215+
]
216+
},
217+
{
218+
"cell_type": "markdown",
219+
"id": "7af57281",
220+
"metadata": {},
221+
"source": [
222+
"- Start by manually testing how the DHT works: from any peer (e.g. ID 42), perform some searches (`search`) and retrieve some content (`retrieve_page`). Do not hesitate to test partial searches (empty intersection) and unsuccessful searches.\n",
223+
"- Programmatically, retrieve from a peer the content of 2000 pages among those published (you can use the list of titles you have saved). What is the largest number of articles indexed by a peer?"
224+
]
225+
}
226+
],
227+
"metadata": {
228+
"kernelspec": {
229+
"display_name": "Python 3 (ipykernel)",
230+
"language": "python",
231+
"name": "python3"
232+
},
233+
"language_info": {
234+
"codemirror_mode": {
235+
"name": "ipython",
236+
"version": 3
237+
},
238+
"file_extension": ".py",
239+
"mimetype": "text/x-python",
240+
"name": "python",
241+
"nbconvert_exporter": "python",
242+
"pygments_lexer": "ipython3",
243+
"version": "3.11.4"
244+
},
245+
"toc": {
246+
"base_numbering": 1,
247+
"nav_menu": {},
248+
"number_sections": true,
249+
"sideBar": true,
250+
"skip_h1_title": true,
251+
"title_cell": "Table of Contents",
252+
"title_sidebar": "Contents",
253+
"toc_cell": false,
254+
"toc_position": {},
255+
"toc_section_display": true,
256+
"toc_window_display": false
257+
}
258+
},
259+
"nbformat": 4,
260+
"nbformat_minor": 5
261+
}

0 commit comments

Comments
 (0)