- New E5 models added to model registry (#568). E5 V2 and Multilingual E5 models are now available for use. The new E5 V2 models outperform their E5 counterparts in the BEIR benchmark, as seen here. See all available models here.
- Dockerfile optimisation (#569). A pre-built Marqo base image results in reduced image layers and increased build speed, meaning neater docker pulls and an overall better development experience.
- Major README overhaul (#573). The README has been revamped with up-to-date examples and easier to follow instructions.
- New security policy (#574).
- Improved testing pipeline (#582 & #586). Tests now trigger on pull request updates. This results in safer and easier merges to mainline.
- Updated requirements files. Now the
requirements.dev.txtshould be used to install requirements for development environments (#569). Version pins forprotobuf&onnxhave been removed while a version pin foranyiohas been added (#581, & #589). - General readability improvements (#577, #578, #587, & #580)
- A huge thank you to all our 3.5k stargazers!
- Shoutout to @vladdoster for all the useful spelling and grammar edits!
- Thanks everyone for keeping our forum bustling. Don't hesitate to keep posting your insights, questions, and feedback!
-
Storage status in health check endpoint (#555 & #559). The
GET /indexes/{index-name}/healthendpoint'sbackendobject will now return the booleanstorage_is_available, to indicate if there is remaining storage space. If space is not available, health status will now returnyellow. See here for detailed usage. -
Score Modifiers search optimization (#566). This optimization reduces latency for searches with the
score_modifiersparameter when field names or weights are changed. See here for detailed usage.
- Improved error message for full storage (#555 & #559). When storage is full, Marqo will return
400 Bad Requestinstead of429 Too Many Requests. - Searching with a zero vector now returns an empty list instead of an internal error (#562).
- A huge thank you to all our 3.3k stargazers!
- Thank you for all the continued discussion in our forum. Keep all the insights, questions, and feedback coming!
-
New field
numberOfVectorsin theget_statsresponse object (#553). This field counts all vectors from all documents in a given index. See here for detailed usage. -
New per-index health check endpoint
GET /indexes/{index-name}/health(#552). This replaces the cluster-level health check endpoint,GET /health, which is deprecated and will be removed in Marqo 2.0.0. See here for detailed usage.
-
Improved image download validation and resource management (#551). Image downloading in Marqo is more stable and resource-efficient now.
-
Adding documents now returns an error when
tensorFieldsis not specified explicitly (#554). This prevents users accidentally creating unwanted tensor fields.
- Thank you for the vibrant discussion in our forum. We love hearing your questions and about your use cases.
-
New parameter
tensor_fieldswill replacenon_tensor_fieldsin theadd_documentsendpoint (#538). Only fields intensor_fieldswill have embeddings generated, offering more granular control over which fields are vectorised. See here for the full list ofadd_documentsparameters and their usage. Thenon_tensor_fieldsparameter is deprecated and will be removed in a future release. Calls toadd_documentswith neither of these parameters specified will now fail. -
Multiple tensor field optimisation (#530). This optimisation results in faster and more stable searches across multiple tensor fields. Please note that indexed documents will now have a different internal document structure, so documents indexed with previous Marqo versions cannot be searched with this version, and vice versa.
-
The
add_documentsendpoint's request body is now an object, with the list of documents under thedocumentskey (#535). The query parametersuse_existing_tensors,image_download_headers,model_auth, andmappingshave been moved to the body as optional keys, and support for these parameters in the query string is deprecated. This change results in shorter URLs and better readability, as values for these parameters no longer need to be URL-encoded. See here for the newadd_documentsAPI usage. Backwards compatibility is supported at the moment but will be removed in a future release. -
Better validation for index creation with custom models (#530). When creating an index with a
modelnot in the registry, Marqo will check ifmodel_propertiesis specified with a properdimension, and raise an error if not. See here for a guide on using custom models. This validation is now done at index creation time, rather than at add documents or search time. -
Stricter
filter_stringsyntax forsearch(#530). Thefilter_stringparameter must have special Lucene characters escaped with a backslash (\) to filter as expected. This will affect filtering on field names or content that contains special characters. See here for more information on special characters and see here for a guide on using Marqo filter strings. -
Removed server-side batching (
batch_sizeparameter) for theadd_documentsendpoint (#527). Instead, client-side batching is encouraged (useclient_batch_sizeinstead ofserver_batch_sizein the python client).
- Multi-field pagination (#530). The
offsetparameter insearchcan now be used to paginate through results spanning multiplesearchable_attributes. This works for bothTENSORandLEXICALsearch. See here for a guide on pagination. - Optimised default index configuration (#540).
- Removed or updated all references to outdated features in the examples and the README (#529).
- Enhanced bulk search test stability (#544).
- Thank you to our 3.2k stargazers!
- We've finally come to our first major release, Marqo 1.0.0! Thanks to all our users and contributors, new and old, for your feedback and support to help us reach this huge milestone. We're excited to continue building Marqo with you. Happy searching!
- Telemetry. Marqo now includes various timing metrics for the
search,bulk_searchandadd_documentsendpoints when the query parametertelemetry=Trueis specified (#506). The metrics will be returned in the response body and provide a breakdown of latencies for various stages of the API call. - Consolidate default device to CUDA when available (#508). By default, Marqo now uses CUDA devices for search and indexing if available. See here for more information. This helps ensure you get the best indexing and search experience without having to explicitly add the device parameter to search and add_documents calls.
- Model download integrity verification (#502). Model files are validated and removed if corrupted during download. This helps ensure that models are not loaded if they are corrupted.
- Remove deprecated
add_or_update_documentsendpoint (#517). - Disable automatic index creation. Marqo will no longer automatically create an index if it does not exist (#516). Attempting to add documents to a non-existent index will now result in an error. This helps provide more certainty about the properties of the index you are adding documents to, and also helps prevent accidental indexing to the wrong index.
- Remove parallel indexing (#523). Marqo no longer supports server-side parallel indexing. This helps deliver a more stable and efficient indexing experience. Parallelisation can still be implemented by the user.
- Improve error messages (#494, #499).
- Improve API request validation (#495).
- Add new multimodal search example (#503).
- Remove autocast for CPU to speed up vectorisation on ARM64 machines (#491).
- Enhance test stability (#514).
- Ignore
.kibanaindex (#512). - Improve handling of whitespace when indexing documents (#521).
- Update CUDA version to 11.4.3 (#525).
- Thank you to our 3.1k stargazers!
-
Load custom SBERT models from cloud storage with authentication (#474). Marqo now supports fetching your fine-tuned public and private SBERT models from Hugging Face and AWS s3. Learn more about using your own SBERT model here. For instructions on loading a private model using authentication, check model auth during search and model auth during add_documents.
-
Bulk search score modifier and context vector support (#469). Support has been added for score modifiers and context vectors to our bulk search API. This can help enhance throughput and performance for certain workloads. Please see documentation for usage.
- A special thank you to our 3.0k stargazers!
- Custom model pre-loading (#475). Public CLIP and OpenCLIP models specified by URL can now be loaded on Marqo startup via the
MARQO_MODELS_TO_PRELOADenvironment variable. These must be formatted as JSON objects withmodelandmodel_properties. See here (configuring pre-loaded models) for usage.
- Fixed arm64 build issue caused by package version conflicts (#478)
- Model authorisation(#460). Non-public OpenCLIP and CLIP models can now be loaded
from Hugging Face and AWS s3 via the
model_locationsettings object andmodel_auth. See here (model auth during search) and here (model auth during add_documents) for usage. - Max replicas configuration (#465). Marqo admins now have more control over the max number of replicas that can be set for indexes on the Marqo instance. See here for how to configure this.
- Marqo now allows for a maximum of 1 replica per index by default (#465).
- README improvements (#468)
- OpenCLIP version bumped (#461)
- Added extra tests (#464)
- Unneeded files are now excluded in Docker builds (#448, #426)
- Thank you to our 2.9k stargazers!
- Thank you to community members for the increasingly exciting discussions on our Slack channel. Feedback, questions and hearing about use cases helps us build a great open source product.
- Thank you to @jalajk24 for the PR to exclude unneeded files from Docker builds!
- New E5 model type is available (#419). E5 models are state of the art general-purpose text embedding models that obtained the best results on the MTEB benchmark when released in Dec 2022. Read more about these models here.
- Automatic model ejection (#372). Automatic model ejection helps prevent out-of-memory (OOM) errors on machines with a larger amount of CPU memory (16GB+) by ejecting the least recently used model.
- Speech processing article and example (#431). @OwenPendrighElliott demonstrates how you can build and query a Marqo index from audio clips.
- Delete optimisation (#436). The
/deleteendpoint can now handle a higher volume of requests. - Inference calls can now execute in batches, with batch size configurable by an environment variable (#376).
- Configurable max value validation for HNSW graph parameters (#424). See here for how to configure.
- Configurable maximum number of tensor search attributes (#430). See here for how to configure.
- Unification of vectorise output type (#432)
- Improved test pipeline reliability (#438, #439)
- Additional image download tests (#402, #442)
- Minor fix in the Iron Manual example (#440)
- Refactored HTTP requests wrapper (#367)
- Thank you to our 2.8k stargazers!
- Thank you community members raising issues and discussions in our Slack channel.
- Thank you @jess-lord and others for raising issues
- New parameters that allow tweaking of Marqo indexes' underlying HNSW graph.
ef_constructionandmcan be defined at index time (#386, #420, #421), giving you more control over the relevancy/speed tradeoff. See usage and more details here. - Score modification fields (#414). Rank documents using knn similarity in addition to document metadata ( #414). This allows integer or float fields from a document to bias a document's score during the knn search and allows additional ranking signals to be used. Use cases include giving more reputable documents higher weighting and de-duplicating search results. See usage here.
- Added validation for unknown parameters during bulk search (#413).
- Improved concurrency handling when adding documents to an index as it's being deleted (#407).
- Better error messages for multimodal combination fields (#395).
- Examples of recently added features added to README (#403).
- Thank you to our 2.6k stargazers.
- Thank you to @anlrde, @strich, @feature-hope, @bazuker for raising issues!
- Bulk search (#363, #373). Conduct multiple searches with just one request. This improves search throughput in Marqo by parallelising multiple search queries in a single API call. The average search time can be decreased up to 30%, depending on your devices and models. Check out the usage guide here
- Configurable number of index replicas (#391).
You can now configure how many replicas to make for an index in Marqo using the
number_of_replicasparameter. Marqo makes 1 replica by default. We recommend having at least one replica to prevent data loss. See the usage guide here - Use your own vectors during searches (#381). Use your own vectors as context for your queries. Your vectors will be incorporated into the query using a weighted sum approach, allowing you to reduce the number of inference requests for duplicated content. Check out the usage guide here
- Fixed a bug where some Open CLIP models were unable to load checkpoints from the cache (#387).
- Fixed a bug where multimodal search vectors are not combined based on expected weights (#384).
- Fixed a bug where multimodal document vectors are not combined in an expected way.
numpy.sumwas used rather thannumpy.mean. (#384). - Fixed a bug where an unexpected error is thrown when
using_existing_tensor = Trueand documents are added with duplicate IDs (#390). - Fixed a bug where the index settings validation did not catch the
modelfield if it is in the incorrect part of the settings json (#365). - Added missing descriptions and requirement files on our GPT-examples (#349).
- Updated the instructions to start Marqo-os (#371).
- Improved the Marqo start-up time by incorporating the downloading of the punkt tokenizer into the dockerfile (#346).
- Thank you to our 2.5k stargazers.
- Thank you to @ed-muthiah for submitting a PR (#349) that added missing descriptions and requirement files on our GPT-examples.
- Multimodal tensor combination (#332, #355). Combine image and text data into a single vector! Multimodal combination objects can be added as Marqo document fields. For example, this can be used to encode text metadata into image vectors. See usage here.
- Fixed a bug that prevented CLIP's device check from behaving as expected (#337)
- CLIP utils is set to use the OpenCLIP default tokenizer so that long text inputs are truncated correctly (#351).
- Thank you to our 2.4k stargazers
- Thank you to @ed-muthiah, @codebrain and others for raising issues.
use_existing_tensorsflag, foradd_documents(#335). Use existing Marqo tensors to autofill unchanged tensor fields, for existing documents. This lets you quickly add new metadata while minimising inference operations. See usage here.image_download_headersparameter forsearchandadd_documents(#336). Index and search non-publicly available images. Add image download auth information toadd_documentsandsearchrequests. See usage here.
- The index cache is now updated on intervals of 2 seconds (#333), rather than on every search. This reduces the pressure on Marqo-OS, allowing for greater search and indexing throughput.
- Helpful validation errors for invalid index settings (#330). Helpful error messages allow for a smoother getting-started experience.
- Automatic precision conversion to
fp32when usingfp16models on CPU (#331). - Broadening of the types of image download errors gracefully handled. (#321)
- Support for custom CLIP models using the OpenAI and OpenCLIP architectures (#286). Read about usage here.
- Concurrency throttling (#304). Configure the number of allowed concurrent indexing and search threads. Read about usage here.
- Configurable logging levels (#314). Adjust log output for your debugging/log storage needs. See how to configure log level here.
- New array datatype (#312). You can use these arrays as a collection of tags to filter on! See usage here.
- Boost tensor fields during search (#300). Weight fields as higher and lower relative to each other during search. Use this to get a mix of results that suits your use case. See usage here.
- Weighted multimodal queries (#307). You can now search with a dictionary of weighted queries. If searching an image index, these queries can be a weighted mix of image URLs and text. See usage here.
- New GPT-Marqo integration example and article. Turn your boring user manual into a question-answering bot, with an optional persona, with GPT + Marqo!
- Added new OpenCLIP models to Marqo (#299)
- Concurrent image downloads (#281, #311)
- Blazingly fast
fp16ViT CLIP models (#286). See usage here - Reduction of data transfer between Marqo and Marqo-os (#300)
- We see a 3.0x indexing speedup, and a 1.7x search speedup, using the new
fp16/ViT-L/14CLIP model, compared to the previous release usingViT-L/14.
- Fixed 500 error when creating an index while only specifying
number_of_shards(#293) - Fixed model cache management no parsing reranker model properties properly (#308)
- Thank you to our 2.3k stargazers
- Thank you to @codebrain and others for raising issues.
- Multilingual CLIP (#267). Search images in the language you want! Marqo now incorporates open source multilingual CLIP models. A list of available multilingual CLIP models are available here.
- Exact text matching (#243, #288). Search for specific words and phrases using double quotes (
" ") in lexical search. See usage here.
- Search speed-up (#278). Latency reduction from Marqo-os indexes reconfigurations.
Thank you to our 2.2k stargazers and 80+ forkers!
- Pagination (#251). Navigate through pages of results. Provide an extensive end-user search experience without having to keep results in memory! See usage here
- The
/modelsendpoint (#239). View what models are loaded, and on what device. This lets Marqo admins examine loaded models and prune unneeded ones. See usage here - The
/deviceendpoint (#239). See resource usage for the machine Marqo is running on. This helps Marqo admins manage resources on remote Marqo instances. See usage here - The index settings endpoint (
/indexes/{index_name}/settings)(#248). See the model and parameters used by each index. See usage here. - Latency log outputs (#242). Marqo admins have better transparency about the latencies for each step of the Marqo indexing and search request pipeline
- ONNX CLIP models are now available (#245). Index and search images in Marqo with CLIP models in the faster, and open, ONNX format - created by Marqo's machine learning team. These ONNX CLIP models give Marqo up to a 35% speedup over standard CLIP models. These ONNX CLIP models are open sourced by Marqo. Read about usage here.
- New simple image search guide (#253, #263).
- ⭐️ We've just hit over 2.1k GitHub stars! ⭐️ So an extra special thanks to our stargazers and contributors who make Marqo possible.
- Generic model support (#179). Create an index with your favourite SBERT-type models from HuggingFace! Read about usage here
- Visual search update 2. (#214). Search-time image reranking and open-vocabulary localization, based on users' queries, is now available with the Owl-ViT model. Locate the part of the image corresponding to your query! Read about usage here
- Visual search update 1. (#214). Better image patching. In addition to faster-rcnn, you can now use yolox or attention based (DINO) region proposal as a patching method at indexing time. This allows localization as the sub patches of the image can be searched. Read about usage here.
Check out this article about how this update makes image search awesome.
- Fixed imports and outdated Python client usage in Wikipedia demo (#216)
- Thank you to @georgewritescode for debugging and updating the Wikipedia demo
- Thank you to our 1.8k stargazers and 60+ forkers!
- Set k to limit to for Marqo-os search queries (#219)
- Reduced the amount of metadata returned from Marqo-os, on searches (#218)
- Set default kNN m value to 16 (#222)
- Better error messages when downloading an image fails (#198)
- Bug where filtering wouldn't work on fields with spaces (#213), resolving #115
- Get indexes endpoint:
GET /indexes(#181). Use this endpoint to inspect existing Marqo indexes. Read about usage here. - Non-tensor fields(#161). During the indexing phase, mark fields as non-tensor to prevent tensors being created for them. This helps speed up indexing and reduce storage for fields where keyword search is good enough. For example: email, name and categorical fields. These fields can still be used for filtering. Read about usage here.
- Configurable preloaded models(#155). Specify which machine learning model to load as Marqo starts. This prevents a delay during initial search and index commands after Marqo starts. Read about usage here.
- New example and article: use Marqo to provide context for up-to-date GPT3 news summary generation (#171, #174). Special thanks to @iain-mackie for this example.
- Updated developer guide (#164)
- Updated requirements which prevented Marqo being built as an arm64 image (#173)
- Backend updated to use marqo-os:0.0.3 (#183)
- Default request timeout has been increased from 2 to 75 seconds (#184)
- For work on the GPT3 news summary generation example: @iain-mackie
- For contributing the non-tensor fields feature: @jeadie
- Thank you to our users who raise issues and give us valuable feeback
- Thank you to our 1.4k+ star gazers and 50+ forkers!
- 429 (too many request errors) are propagated from Marqo-os to the user properly (#150)
- Health check endpoint:
GET /health. An endpoint that can be used to inspect the status of Marqo and Marqo's backend (Marqo-os) (#128). Read about usage here. - Marqo can be launched with environment variables that define limits around maximum number of fields per index, maximum document size and the maximum number of documents that can be retrieved (#135). Read about usage here.
- README translations:
- Chinese 🇨🇳 (by @wanliAlex, #133)
- Polish 🇵🇱 (by @MichalLuck, #136)
- Ukrainian 🇺🇦 (by @dmyzlata, #138)
- French 🇫🇷 (by @rym-oualha, #147)
- The home
/json response has been updated. If you have logic that reads the endpoint root, please update it (#128). - The Python client's
add_documents()andupdate_documents()batch_sizeparameter has been replaced byserver_batch_sizeandclient_batch_sizeparameters (py-marqo#27), (py-marqo#28)
- Each text field just creates a top level Marqo-os text field, without any keywords (#135)
- Very large fields get their tensor_facet keywords ignored, rather than Marqo-OS preventing the doc being indexed (#135)
- Tensor facets can no longer have _id as a filtering field (#135)
- FastAPI runs with better concurrency (#128)
- Get documents by IDs and lexical search and no longer returns vectors if expose_facets isn't specified
- Fixed batching bug in Python client (py-marqo#28)
- If a large request to add_documents or update_documents results in a document adding fields such that the index field limit is exceeded, the entire operation will fail (without resilience). Mitigate this sending
add_documentsandupdate_documentsrequests with smaller batches of documents. - For optimal indexing of large volumes of images, we recommend that the images are hosted on the same region and cloud provider as Marqo.
- For their translation work: @rym-oualha, @dmyzlata, @wanliAlex, @dmyzlata, @MichalLuck
- For raising issues and helping with READMEs: @kdewald, @llermaly, @namit343
- Thank you to our 900+ star gazers and 30+ forkers
Added Open CLIP models and added features to the get document endpoint.
- Added Open CLIP models (#116). Read about usage here
- Added the ability to get multiple documents by ID (#122). Read about usage here
- Added the ability to get document tensor facets through the get document endpoint (#122). Read about usage here
Adding the attributesToRetrieve to the search endpoint and added the update documents endpoints