Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ellipsize glob paths similar to https://github.com/Eventual-Inc/Daft/pull/2695 #2709

Closed
universalmind303 opened this issue Aug 22, 2024 · 0 comments · Fixed by #2809
Labels
good first issue Good for newcomers

Comments

@universalmind303
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
if you create a plan that uses many dynamically generated urls, then run explain, the output is a massive wall of text littered with many urls, making it nearly illegible

import daft

urls = [f"https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-{n:05d}.parquet" for n in range(1, 500)] 

df = daft.read_parquet(urls)
df.explain(show_all=True)
== Unoptimized Logical Plan ==
  • GlobScanOperator
    | Glob paths = [https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00001.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00002.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00003.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00004.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00005.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00006.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00007.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00008.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00009.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00010.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00011.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00012.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00013.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00014.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00015.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00016.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00017.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00018.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00019.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00020.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00021.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00022.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00023.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00024.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00025.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00026.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00027.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00028.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00029.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00030.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00031.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00032.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00033.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00034.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00035.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00036.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00037.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00038.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00039.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00040.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00041.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00042.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00043.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00044.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00045.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00046.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00047.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00048.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00049.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00050.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00051.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00052.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00053.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00054.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00055.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00056.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00057.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00058.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00059.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00060.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00061.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00062.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00063.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00064.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00065.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00066.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00067.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00068.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00069.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00070.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00071.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00072.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00073.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00074.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00075.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00076.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00077.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00078.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00079.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00080.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00081.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00082.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00083.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00084.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00085.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00086.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00087.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00088.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00089.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00090.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00091.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00092.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00093.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00094.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00095.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00096.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00097.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00098.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00099.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00100.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00101.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00102.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00103.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00104.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00105.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00106.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00107.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00108.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00109.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00110.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00111.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00112.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00113.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00114.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00115.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00116.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00117.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00118.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00119.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00120.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00121.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00122.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00123.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00124.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00125.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00126.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00127.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00128.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00129.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00130.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00131.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00132.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00133.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00134.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00135.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00136.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00137.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00138.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00139.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00140.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00141.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00142.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00143.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00144.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00145.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00146.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00147.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00148.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00149.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00150.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00151.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00152.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00153.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00154.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00155.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00156.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00157.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00158.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00159.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00160.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00161.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00162.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00163.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00164.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00165.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00166.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00167.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00168.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00169.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00170.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00171.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00172.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00173.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00174.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00175.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00176.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00177.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00178.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00179.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00180.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00181.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00182.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00183.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00184.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00185.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00186.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00187.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00188.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00189.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00190.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00191.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00192.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00193.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00194.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00195.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00196.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00197.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00198.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00199.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00200.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00201.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00202.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00203.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00204.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00205.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00206.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00207.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00208.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00209.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00210.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00211.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00212.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00213.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00214.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00215.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00216.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00217.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00218.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00219.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00220.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00221.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00222.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00223.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00224.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00225.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00226.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00227.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00228.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00229.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00230.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00231.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00232.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00233.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00234.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00235.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00236.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00237.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00238.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00239.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00240.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00241.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00242.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00243.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00244.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00245.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00246.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00247.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00248.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00249.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00250.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00251.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00252.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00253.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00254.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00255.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00256.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00257.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00258.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00259.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00260.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00261.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00262.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00263.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00264.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00265.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00266.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00267.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00268.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00269.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00270.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00271.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00272.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00273.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00274.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00275.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00276.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00277.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00278.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00279.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00280.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00281.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00282.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00283.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00284.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00285.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00286.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00287.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00288.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00289.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00290.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00291.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00292.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00293.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00294.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00295.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00296.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00297.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00298.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00299.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00300.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00301.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00302.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00303.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00304.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00305.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00306.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00307.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00308.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00309.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00310.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00311.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00312.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00313.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00314.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00315.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00316.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00317.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00318.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00319.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00320.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00321.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00322.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00323.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00324.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00325.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00326.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00327.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00328.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00329.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00330.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00331.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00332.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00333.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00334.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00335.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00336.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00337.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00338.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00339.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00340.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00341.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00342.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00343.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00344.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00345.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00346.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00347.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00348.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00349.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00350.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00351.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00352.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00353.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00354.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00355.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00356.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00357.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00358.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00359.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00360.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00361.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00362.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00363.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00364.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00365.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00366.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00367.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00368.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00369.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00370.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00371.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00372.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00373.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00374.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00375.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00376.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00377.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00378.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00379.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00380.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00381.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00382.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00383.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00384.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00385.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00386.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00387.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00388.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00389.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00390.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00391.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00392.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00393.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00394.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00395.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00396.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00397.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00398.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00399.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00400.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00401.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00402.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00403.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00404.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00405.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00406.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00407.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00408.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00409.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00410.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00411.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00412.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00413.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00414.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00415.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00416.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00417.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00418.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00419.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00420.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00421.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00422.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00423.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00424.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00425.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00426.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00427.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00428.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00429.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00430.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00431.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00432.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00433.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00434.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00435.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00436.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00437.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00438.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00439.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00440.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00441.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00442.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00443.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00444.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00445.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00446.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00447.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00448.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00449.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00450.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00451.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00452.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00453.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00454.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00455.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00456.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00457.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00458.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00459.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00460.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00461.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00462.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00463.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00464.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00465.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00466.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00467.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00468.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00469.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00470.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00471.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00472.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00473.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00474.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00475.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00476.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00477.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00478.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00479.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00480.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00481.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00482.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00483.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00484.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00485.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00486.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00487.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00488.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00489.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00490.parquet, https://huggingface.co/datasets/bigdata-
    | pw/Flickr/resolve/main/part-00491.parquet, https://huggingface.co/datasets/
    | bigdata-pw/Flickr/resolve/main/part-00492.parquet, https://huggingface.co/
    | datasets/bigdata-pw/Flickr/resolve/main/part-00493.parquet, https://
    | huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-00494.parquet,
    | https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/part-
    | 00495.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/main/
    | part-00496.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/resolve/
    | main/part-00497.parquet, https://huggingface.co/datasets/bigdata-pw/Flickr/
    | resolve/main/part-00498.parquet, https://huggingface.co/datasets/bigdata-pw/
    | Flickr/resolve/main/part-00499.parquet]
    | Coerce int96 timestamp unit = Nanoseconds
    | IO config = S3 config = { Max connections = 8, Retry initial backoff ms =
    | 1000, Connect timeout ms = 30000, Read timeout ms = 30000, Max retries = 25,
    | Retry mode = adaptive, Anonymous = false, Use SSL = true, Verify SSL = true,
    | Check hostname SSL = true, Requester pays = false, Force Virtual Addressing =
    | false }, Azure config = { Use Fabric Endpoint = false, Anonymous = false, Use
    | SSL = true }, GCS config = { Anonymous = false }, HTTP config = { user_agent
    | = daft/0.0.1 }
    | Use multithreading = true
    | File schema = id#Utf8, owner#Utf8, url_sq#Utf8, width_sq#Int32, height_sq#Int32,
    | url_q#Utf8, width_q#Int32, height_q#Int32, url_t#Utf8, width_t#Int32,
    | height_t#Int32, url_s#Utf8, width_s#Int32, height_s#Int32, url_n#Utf8,
    | width_n#Int32, height_n#Int32, url_w#Utf8, width_w#Int32, height_w#Int32,
    | url_m#Utf8, width_m#Int32, height_m#Int32, url_z#Utf8, width_z#Int32,
    | height_z#Int32, url_c#Utf8, width_c#Int32, height_c#Int32, url_l#Utf8,
    | width_l#Int32, height_l#Int32, url_h#Utf8, width_h#Int32, height_h#Int32,
    | url_k#Utf8, width_k#Int32, height_k#Int32, url_3k#Utf8, width_3k#Int32,
    | height_3k#Int32, url_4k#Utf8, width_4k#Int32, height_4k#Int32, url_5k#Utf8,
    | width_5k#Int32, height_5k#Int32, url_6k#Utf8, width_6k#Int32, height_6k#Int32,
    | url_o#Utf8, o_width#Utf8, o_height#Utf8, title#Utf8, description#Utf8,
    | tags#Utf8, latitude#Utf8, longitude#Utf8, accuracy#Utf8, place_id#Utf8,
    | woeid#Utf8, datetaken#Utf8, dateupload#Utf8, dateadded#Utf8, count_views#Utf8,
    | count_faves#Utf8, count_comments#Utf8, license#Int32, license_name#Utf8,
    | safe#Int32, safety_level#Utf8, rotation#Int32, originalformat#Utf8,
    | content_type#Utf8, media#Utf8, machine_tags#Utf8, sizes#Utf8
    | Partitioning keys = []
    | Output schema = id#Utf8, owner#Utf8, url_sq#Utf8, width_sq#Int32,
    | height_sq#Int32, url_q#Utf8, width_q#Int32, height_q#Int32, url_t#Utf8,
    | width_t#Int32, height_t#Int32, url_s#Utf8, width_s#Int32, height_s#Int32,
    | url_n#Utf8, width_n#Int32, height_n#Int32, url_w#Utf8, width_w#Int32,
    | height_w#Int32, url_m#Utf8, width_m#Int32, height_m#Int32, url_z#Utf8,
    | width_z#Int32, height_z#Int32, url_c#Utf8, width_c#Int32, height_c#Int32,
    | url_l#Utf8, width_l#Int32, height_l#Int32, url_h#Utf8, width_h#Int32,
    | height_h#Int32, url_k#Utf8, width_k#Int32, height_k#Int32, url_3k#Utf8,
    | width_3k#Int32, height_3k#Int32, url_4k#Utf8, width_4k#Int32, height_4k#Int32,
    | url_5k#Utf8, width_5k#Int32, height_5k#Int32, url_6k#Utf8, width_6k#Int32,
    | height_6k#Int32, url_o#Utf8, o_width#Utf8, o_height#Utf8, title#Utf8,
    | description#Utf8, tags#Utf8, latitude#Utf8, longitude#Utf8, accuracy#Utf8,
    | place_id#Utf8, woeid#Utf8, datetaken#Utf8, dateupload#Utf8, dateadded#Utf8,
    | count_views#Utf8, count_faves#Utf8, count_comments#Utf8, license#Int32,
    | license_name#Utf8, safe#Int32, safety_level#Utf8, rotation#Int32,
    | originalformat#Utf8, content_type#Utf8, media#Utf8, machine_tags#Utf8,
    | sizes#Utf8

Describe the solution you'd like
Similar to the scan tasks that are ellipsized, we should do the same for urls in the logical plan

Describe alternatives you've considered
None

Additional context
Add any other context or screenshots about the feature request here.

@universalmind303 universalmind303 added the good first issue Good for newcomers label Aug 22, 2024
anmolsingh20 added a commit to anmolsingh20/Daft that referenced this issue Sep 8, 2024
anmolsingh20 added a commit to anmolsingh20/Daft that referenced this issue Sep 9, 2024
samster25 pushed a commit that referenced this issue Sep 9, 2024
Resolves #2709
- Previously if there were multiple urls in the logical plan, it
cluttered the output of 'df.explain' with massive text. Now we add
ellipses if there are more than six, improving readability.
- The current test emulates multiple urls from the same test fixture
'mvp.parquet'. Ideally there should be a test fixture with multiple
parts of a parquet file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant