Multi-source providers

[Tahul]

Multi-source is now possible with @nuxt/content, but is still an experimental feature.

Local sources (same filesystem) are already supported, but remote ones are still a pending task.

Here is a list of providers we could have in mind to support:

• Other GitHub repositories

MDC Syntax would most likely only be available from providers that can return raw text data though.

Here is what package is used to manage content sources, and where should these drivers be implemented:
https://github.com/unjs/unstorage

[pi0]

Most provider sources could be supported by using a custom unstorage driver and fetch. We just need to map each into a Content module compatible format (List, parsed (alternative to markdown AST?) and Meta)

BTW, while this is a nice idea to have CMS support with content style queries, I feel this is kinda out of the content module's scope. Almost each of the providers mentioned above has their unique abilities that could be limited when going behind abstraction and we already have a Nuxt module for a couple of them.

[pi0]

I would separate this issue into:

• Multi-source support (fetching from other repos from the same content format. continuation of How get content from other repos ? #265)
• A discussion to investigate the possibility and benefits of integration with external CMS providers.

[atinux]

Agree to split this issue in two, also I do believe that it is out of the scope of content regarding MDC and other providers personally.

Creating one following #265 shall be enough

[rahulkumarsingh73690]

Adding content sources from other GitHub Repositories is enough because in most cases we need separate docs repository from docs website source code. This feature helps prevents the website from copying.

[Tahul]

These are more ideas for future possible enhancements than real objectives.

I think supporting other GitHub repositories is an evidence, and supporting Notion would be a really nice-to-have feature.

I completely agree for some of these providers, a standalone module would fit the use case better than being in the content module itself.

Still, I think at some point the content module has something to offer to these standalone modules, which is MDC parsing and rendering!

MDC Syntax allows the usage of project components by parsing a syntax that can be stored by any content provider, and I think providing the support for this syntax to these externals providers would be amazing adoption-wise for @nuxt/content.

[ManasMadrecha]

Local sources (same filesystem) are already supported

Kindly add some easy-to-understand documentation on how to fetch from different folders on same PC. The current documentation on using drivers and all is confusing.

• What are drivers? What are the driver options? Where to find list of such options? What is resolve?
• What's the difference between the name, prefix, and source dir in the base? And why to have different names for them?
• How will fetching from private GitHub repo work? How to authenticate it? Or will it even work?

---

The reason I started using Content module (and Nuxt) is because it is very easy to learn and understand, even for non-coders. The whole Content website is very simple. So, whenever this multi-source feature becomes fully functional, kindly simplify this part of the documentation as well, especially as if oriented towards a novice level, and not for people who already know this stuff (because you all already do 😁).

Thanks.

[atinux]

Mutli source is next in our roadmap, it was not a blocker for the initial release.

We do need it since we plan to have all of our docs in one website 😊

[ManasMadrecha]

@Tahul, @atinux Kindly help by filling these blanks 😄

My Nuxt Content app is in this folder C:\Users\manas\OneDrive\Projects\nuxt3-content2 on my PC.

What base config should be added in sources object for the following 3 types of folders (apart from the existing content folder)?

1. ✅ SOLVED: If another folder content-2 present in the same folder as the app at the root level?

Issue

- Simply adding another item in the `sources` as string `content-2` works, and even fetching works properly. - But to add `prefix` to it, we need an Object syntax inside the `sources` item, for which adding `driver: fs` and `base` properties is compulsory. - So, what should be `base` for such folder?

{
    name: 'content-2',
    prefix: '/f2', 
    driver: 'fs',
    base: path.resolve('.', 'content-2') // ~~What should come here?~~ Never mind, got it. 
}

👍🏻 SOLVED

So, the dot '.' represents the root.

Also, the name automatically becomes the value of the _source key inside the file's frontmatter. This need not be the name of the folder where the md files are stored. This is simply just another frontmatter key. This should be documented on the website. 😁 Can we Query the files using this _source key inside .where options? Most probably, but haven't tried yet.

Note to maintainers: Kindly add in the documentation that name becomes _source key in frontmatter.

2. ✅ SOLVED: If a folder content-2 is present in the same app's folder, but nested inside some other folder?

Issue

• C:\Users\manas\OneDrive\Projects\nuxt3-content2\docs\content-2
• I don't want the whole docs folder to be considered for the Content Module, but just the content-2 child folder present in it.

{
    name: 'content-2 inside docs',
    prefix: '/docsc2', 
    driver: 'fs',
    base: path.resolve('.','docs/content-2') // ~~What should come here?~~ Never mind, got it. 
}

👍🏻 SOLVED

From the root ., now you can add any path easily.

3. ✅ SOLVED: If another folder Daily is present elsewhere on the computer?

Issue

• This new folder is present in C:\Users\manas\OneDrive\Documents\Obsidian Vaults\Master.
• This folder contains other file types as well, apart from markdown files.

{
    name: 'Daily', // ~~Is space allowed in the folder name here?~~ Yes, it just gets copied as frontmatter key.
    prefix: '/daily', // This is main. It is what is used to query the posts.
    driver: 'fs',
    base: path.resolve('C:/Users/manas/OneDrive/Documents/Obsidian Vaults/Master','Daily') 
            ~~// What should come here?~~ 
                // Never mind, got it.
}

👍🏻 SOLVED

Even on Windows, use / in path, instead of default \. https://stackoverflow.com/questions/31847712/node-js-fs-module-and-windows-paths.

😮Another issue of Folder with images

While base issue is solved, but because the Daily folder contains images as well, this is what I get in the Terminal. And even the page takes so much time to reload that fetching doesn't work takes half a minute.

Why doesn't the Nuxt Content module simply exclude these file types? So, creating a new issue for it.

---

Thanks in advance. With these three questions solved, it will be very easy for future users (non-coders) as well. ❤️

[rahulkumarsingh73690]

Any update on this idea?

[augnustin]

I'm also looking for a way to fetch remote data.

Most provider sources could be supported by using a custom unstorage driver and fetch.

This would definitely fit my need. Any chance to get a working example on this? Even if this is done by hand first.

Happy to help

[ricardoaguiar]

I'm also looking to fetch data from different directories, and have a fallback if the content directory is empty.
For example, if the <content> directory is empty, we could have a fallback <another-content-directory> that will always have a default content in it.
That could be useful when working with multiple repos coming from a single upstream.
Glad to help.

[atinux]

The fallback strategy is pretty tricky to do actually @ricardoaguiar but is interesting.

Do you have a reproduction to test?

[adamdehaven]

Could we get an example of just how to use the existing http driver to fetch documents from an API? Would be super-helpful.