Use Transformer to perform Cross encoder

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Background\n\nSentence transformer have packaged the cross-encoder class for cross encoders. Sometimes we do not want import sentence transformer but still need some of the functions of cross-encoder, we can use transformer to achieve the same.  This is not remaking wheels,  just a practise","metadata":{}},{"cell_type":"markdown","source":"## Prepare","metadata":{}},{"cell_type":"code","source":"!pip install transformers==4.39.2\n!pip install nltk==3.8.1","metadata":{"execution":{"iopub.status.busy":"2024-05-20T15:08:52.795083Z","iopub.execute_input":"2024-05-20T15:08:52.795592Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Collecting transformers==4.39.2\n  Downloading transformers-4.39.2-py3-none-any.whl.metadata (134 kB)\n\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (3.13.1)\nRequirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (0.22.2)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (1.26.4)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (21.3)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (6.0.1)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (2023.12.25)\nRequirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (2.31.0)\nRequirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (0.15.2)\nRequirement already satisfied: safetensors>=0.4.1 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (0.4.3)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.10/site-packages (from transformers==4.39.2) (4.66.1)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.39.2) (2024.2.0)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.39.2) (4.9.0)\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->transformers==4.39.2) (3.1.1)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.39.2) (3.3.2)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.39.2) (3.6)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.39.2) (1.26.18)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.39.2) (2024.2.2)\nDownloading transformers-4.39.2-py3-none-any.whl (8.8 MB)\n\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.8/8.8 MB\u001b[0m \u001b[31m77.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n\u001b[?25h","output_type":"stream"}]},{"cell_type":"markdown","source":"### Data preparation","metadata":{}},{"cell_type":"code","source":"from nltk import sent_tokenize\n\nmelbourne_info=\"\"\"Melbourne: A mosaic of stories under the Southern Cross\n\nMelbourne, Australia's cultural gem, shimmers like a kaleidoscope beneath the watchful gaze of the Southern Cross. Its tapestry is woven from threads of history, painted with strokes of vibrant street art, and hums with the rhythm of diverse voices. From the grand bluestone facades of Victorian-era buildings to the glass-and-steel skyscrapers whispering tales of progress, the city breathes a unique blend of old and new.\n\nWander cobbled laneways adorned with stencils and murals, each a silent storyteller chronicling artistic dreams. Duck into hidden bars, their intimate spaces echoing with laughter and clinking glasses as locals unwind over craft beers brewed with Melbourne's signature spirit. Inhale the intoxicating aroma of coffee wafting from countless cafes, fueling the city's creative pulse and fostering lively debates that spill onto sun-drenched terraces.\n\nImmerse yourself in the vibrant tapestry of cultures that call Melbourne home. Explore bustling Chinatown, a sensory feast of sights, sounds, and smells, where dim sum restaurants overflow with families sharing stories over steaming baskets. Meander through vibrant Greek neighborhoods, where tavernas spill onto lively streets filled with the aroma of souvlaki and the sounds of bouzouki music. Unwind in the lush Royal Botanic Gardens, a tranquil oasis where native flora bursts with color and the serene lake reflects the city's ever-changing moods.\n\nStep back in time at Federation Square, where grand buildings whisper tales of the city's colonial past. Marvel at the architectural marvel of St. Paul's Cathedral, its spires reaching towards the sky like silent prayers. Delve into the city's sporting passion at the Melbourne Cricket Ground (MCG), where the roar of the crowd electrifies the air during a thrilling game of cricket or Australian Rules Football.\n\nEmbrace the city's artistic soul at the National Gallery of Victoria, where masterpieces from across the globe come alive. Be mesmerized by contemporary installations at the Australian Centre for the Moving Image (ACMI), or get lost in the whimsical world of children's literature at the Melbourne Museum. As dusk paints the sky in hues of orange and purple, lose yourself in the magic of a performance at the Arts Centre Melbourne, the city's beating heart for all things theatrical.\n\nMelbourne's soul simmers after dark. Laneways transform into open-air bars, pulsating with the energy of live music and animated conversations. Rooftop bars offer breathtaking panoramas of the city bathed in the warm glow of a million lights. Intimate jazz bars tucked away in hidden corners lull you into a state of blissful serenity with soulful melodies.\n\nAs dawn paints the sky with soft pastels, head to Queen Victoria Market, a vibrant labyrinth of stalls overflowing with fresh produce, artisan cheeses, and handcrafted souvenirs. Mingle with the friendly locals, their voices a symphony of accents reflecting the city's multicultural tapestry. Savor a steaming cup of coffee and a flaky pastry at a bustling cafe, the perfect start to a day filled with new discoveries.\n\nMelbourne is more than just a city; it's a feeling. It's the warmth of a smile from a stranger, the shared joy of a spontaneous street performance, the comforting aroma of freshly baked bread wafting from a local bakery. It's the city that embraces individuality, fosters creativity, and celebrates the beauty of diversity. So, come, explore its hidden gems, savor its unique flavors, and immerse yourself in the vibrant melody that is Melbourne.\"\"\"\n\n\nparagraphs=[]\nfor paragraph in melbourne_info.replace('\\r\\n','\\n').split('\\n\\n'):\n    if len(paragraph.strip())>0:\n        paragraphs.append(sent_tokenize(paragraph.strip()))\n\nlen(paragraphs)\n\nwindow_size=3\npassages=[]\nfor paragraph in paragraphs:\n    for start_idx in range(0, len(paragraph), window_size):\n        end_idx=min(start_idx+window_size, len(paragraph))\n        passages.append(' '.join(paragraph[start_idx:end_idx]))\n        \nprint('Paragraphs:', len(paragraphs))\nprint('Sentences:', sum([len(p) for p in paragraphs]))\nprint('Passages:', len(passages))\nprint(passages)","metadata":{"execution":{"iopub.status.busy":"2024-05-20T15:11:01.326512Z","iopub.execute_input":"2024-05-20T15:11:01.326966Z","iopub.status.idle":"2024-05-20T15:11:02.929948Z","shell.execute_reply.started":"2024-05-20T15:11:01.326928Z","shell.execute_reply":"2024-05-20T15:11:02.929012Z"},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"Paragraphs: 9\nSentences: 28\nPassages: 12\n['Melbourne: A mosaic of stories under the Southern Cross', \"Melbourne, Australia's cultural gem, shimmers like a kaleidoscope beneath the watchful gaze of the Southern Cross. Its tapestry is woven from threads of history, painted with strokes of vibrant street art, and hums with the rhythm of diverse voices. From the grand bluestone facades of Victorian-era buildings to the glass-and-steel skyscrapers whispering tales of progress, the city breathes a unique blend of old and new.\", \"Wander cobbled laneways adorned with stencils and murals, each a silent storyteller chronicling artistic dreams. Duck into hidden bars, their intimate spaces echoing with laughter and clinking glasses as locals unwind over craft beers brewed with Melbourne's signature spirit. Inhale the intoxicating aroma of coffee wafting from countless cafes, fueling the city's creative pulse and fostering lively debates that spill onto sun-drenched terraces.\", 'Immerse yourself in the vibrant tapestry of cultures that call Melbourne home. Explore bustling Chinatown, a sensory feast of sights, sounds, and smells, where dim sum restaurants overflow with families sharing stories over steaming baskets. Meander through vibrant Greek neighborhoods, where tavernas spill onto lively streets filled with the aroma of souvlaki and the sounds of bouzouki music.', \"Unwind in the lush Royal Botanic Gardens, a tranquil oasis where native flora bursts with color and the serene lake reflects the city's ever-changing moods.\", \"Step back in time at Federation Square, where grand buildings whisper tales of the city's colonial past. Marvel at the architectural marvel of St. Paul's Cathedral, its spires reaching towards the sky like silent prayers. Delve into the city's sporting passion at the Melbourne Cricket Ground (MCG), where the roar of the crowd electrifies the air during a thrilling game of cricket or Australian Rules Football.\", \"Embrace the city's artistic soul at the National Gallery of Victoria, where masterpieces from across the globe come alive. Be mesmerized by contemporary installations at the Australian Centre for the Moving Image (ACMI), or get lost in the whimsical world of children's literature at the Melbourne Museum. As dusk paints the sky in hues of orange and purple, lose yourself in the magic of a performance at the Arts Centre Melbourne, the city's beating heart for all things theatrical.\", \"Melbourne's soul simmers after dark. Laneways transform into open-air bars, pulsating with the energy of live music and animated conversations. Rooftop bars offer breathtaking panoramas of the city bathed in the warm glow of a million lights.\", 'Intimate jazz bars tucked away in hidden corners lull you into a state of blissful serenity with soulful melodies.', \"As dawn paints the sky with soft pastels, head to Queen Victoria Market, a vibrant labyrinth of stalls overflowing with fresh produce, artisan cheeses, and handcrafted souvenirs. Mingle with the friendly locals, their voices a symphony of accents reflecting the city's multicultural tapestry. Savor a steaming cup of coffee and a flaky pastry at a bustling cafe, the perfect start to a day filled with new discoveries.\", \"Melbourne is more than just a city; it's a feeling. It's the warmth of a smile from a stranger, the shared joy of a spontaneous street performance, the comforting aroma of freshly baked bread wafting from a local bakery. It's the city that embraces individuality, fosters creativity, and celebrates the beauty of diversity.\", 'So, come, explore its hidden gems, savor its unique flavors, and immerse yourself in the vibrant melody that is Melbourne.']\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### Model preparation ","metadata":{}},{"cell_type":"code","source":"from transformers import CrossEncoder\n\nmodel=CrossEncoder('cross-encoder/ms-marco-TinyBERT-L-2')","metadata":{},"execution_count":null,"outputs":[]}]}