Setting up streetmerchant in Unraid via Docker

[dthomas710]

Hello, I'm a huge novice to docker and unraid but I'm learning fast. I appear to be having an issue that no one else has had yet. I'm trying to get streetmerchant installed on my unraid server and it's going extremely poorly. This is because I cannot for the life of me determine what pathway streetmerchant needs to have made for it to be able to install itself into my system. For some reason I cannot find this ANYWHERE, on the github, on the instruction website (which has one step to run on docker....). It's very clear this is the issue based on the logs I'm about to post, so I have no idea why this information isn't easier to find. Probably because I'm a beginner who doesn't understand anything I guess.

I'm going to go step by step through my process so you can see if I'm doing anything wrong.

Step 1) Add container using the link ghcr.io/jef/streetmerchant:latest

Step 2) Open docker container and run npm install

It starts to run like it's going to work:

but then it quickly throws up errors about how it's missing write access and permissions to a number of locations. this is the problem.

" checkPermissions Missing write access to /app/node_modules "

Herein lies the problem. I have tried about ten different possible pathways to get this installed in the proper location where all my other docker containers are installed. That location is: /mnt/user/appdata/ as you can see here, all my other containers are using this location just fine.

For example, here is me attempting to create a /config path for streetmerchant to use

The same error from above is thrown when attempting to run npm install in the container

Other pathways I've tried to give to streetmerchant for it to work:

• /config
• /app (this pathway causes some type of error that makes the container unstartable)
• /app/node_modules
• /app/node_modules/@types (this created a folder called @types in my appdata folder but did not solve the issue)
• both /app/node_modules and /app/node_modules/@types at the same time (this also made container unstartable)
• /streetmerchant
• /appdata
• /pleasework
• /imsodumb

None of these have worked. You can see some of them had a unique effect such as breaking the container entirely, but no pathway that I've chosen has given streetmerchant the ability to create and modify files on my system, which it needs in order to proceed. It tells me to look in a folder for full debugging info but I don't think it has access there either, so no more debugging info can be had. I can't even begin to edit the .env file because I haven't made it that far. I'm hoping that this is just a symptom of me being a novice and not understanding docker, rather than some greater issue.

It really seems like all I need to do is put in the right container path to get this working, and it seems very bizarre to me that I haven't been able to find even a crumb of evidence of what that container path would be..... Any help would be much appreciated

[Zxurian]

I just setup streetmerchant on my system a few days ago. Here are the settings I used. I didn't have to specify any specific paths, only variables specifically related to streetmerchant.

If you want more compatability with Unraid, I also setup the following under Advanced

As far as additional settings, the reference guide on StreetMerchants can be used to create key/value pairs from https://jef.codes/streetmerchant/reference/application/ for configuration like the following

[dthomas710]

Thank you so much for reaching out. I appreciate you attempting to help me. I've followed exactly what you have here down to the letter and it still throws up the exact same error. npm install does not work based on permissions issues, literally no matter what I do. Is this because of I'm missing some type of required environmental variable? None of the variables seem like they are required for the function of this app...... Here is the entire settings page for the docker container:

And here is the error again

What happens when you run npm install on your instance? Does it just work? In that case I have no idea what I'm doing wrong as I've followed your instructions exactly. Could you post a screenshot of your entire container settings page? I may try to clone your variables exactly just to see if I can get this running at all. I'm tearing out my hair here, this really seems like it should be working. I still think I need to map the container path based on the errors I'm seeing here.......

@jef can you please provide assistance as to what I'm doing wrong

[rmp5s]

(posting from phone) So full disclosure, there is an app data folder on unraid that you can go into and edit the env files manually if you want. On the Shares tab of Unraid, one of them will be appdata this share is where docker puts its config / storage data into. There will be one subfolder per docker container. If on the unraid share itself you either export to NFS or share via SMB, that will allow you to browse and modify files within that folder.l directly if you want.

If the container supports env variables, I prefer to set via unraid ui, as. Then it can be saved to the container template and I don't have to worry what file and where needs to be modified.

Nope. There is no "streetmerchant" directory in the appdata share. That was the first place I looked and was like, "hmm..."

Not sure why it's not there.

[dthomas710]

Yup, I'm in the same boat. Mine has created zero files anywhere on my system that I can find. The first place I looked was appdata.

[Zxurian]

did a little research beforehand, so updating my answer from above. (Unraid specific)
If you want a container to write data to a path outside of the container's storage space, then that would have to be specified on the unraid container config itself using a Path configuration entry. This will map a path from the container's space to a path within an unraid share, thus exposing those files outside of the container itself.

Example:

My Bitwarden container writes it'd data to /data within the container. Since that data can't be reset everytime I cycle the docker image, that path is mapped externally to /mnt/user/appdata/bitwarden, which is accessible directly and isn't destroyed when docker shuts down.

Since streetmerchant doesn't write anything to disk, there's no need to map any path configs.

[rmp5s]

did a little research beforehand, so updating my answer from above. (Unraid specific)
If you want a container to write data to a path outside of the container's storage space, then that would have to be specified on the unraid container config itself using a Path configuration entry. This will map a path from the container's space to a path within an unraid share, thus exposing those files outside of the container itself.

Example:

My Bitwarden container writes it'd data to /data within the container. Since that data can't be reset everytime I cycle the docker image, that path is mapped externally to /mnt/user/appdata/bitwarden, which is accessible directly and isn't destroyed when docker shuts down.

Since streetmerchant doesn't write anything to disk, there's no need to map any path configs.

Cool cool. Docker really is a bit of a mind-f in some ways. Coming from the old skool VM world, it definitely takes some getting used to. Hahaha

Now I'm battling NZBGet and the stupid VideoSort script...jeez...what a nightmare. The fun never ends! lol

Thanks for all the help, though! I appreciate it, everyone!

[laurens362]

did a little research beforehand, so updating my answer from above. (Unraid specific)
If you want a container to write data to a path outside of the container's storage space, then that would have to be specified on the unraid container config itself using a Path configuration entry. This will map a path from the container's space to a path within an unraid share, thus exposing those files outside of the container itself.

Example:

My Bitwarden container writes it'd data to /data within the container. Since that data can't be reset everytime I cycle the docker image, that path is mapped externally to /mnt/user/appdata/bitwarden, which is accessible directly and isn't destroyed when docker shuts down.

Since streetmerchant doesn't write anything to disk, there's no need to map any path configs.

Do you happen to know how to edit links to different webstores aswell? Im having a problem with amd site showing me theres stock for the 6000 series while there isnt ( hes rederecting me to a 3950 cpu ) :( on windows i could just edit the store and voila... But on unraid im a bit lost tbh..

[jef]

I'm going to add this to the wiki. Thank you both!

[kiowadriver]

I am setting up street merchant through the unraid "app" channel feed. Here are the instructions to pull down the XML and deploy:
Overview: Support for Docker image streetmerchant in the mitchellriley repo.

Application: streetmerchant - https://jef.codes/streetmerchant/

Docker Hub:https://hub.docker.com/repository/docker/mitchellriley/streetmerchant

GitHub:https://github.com/kiowadriver/unraid-docker/tree/master/mitchellriley

Documentation: https://jef.codes/streetmerchant/

Usage

1. Firstly you need to be running unRAID ver 6.0.0 or later, once installed follow the instructions below:-
2. Navigate to "Docker" tab and then the "Docker Repositories" sub-tab in the unRAID webui
3. Enter in a URL of https://github.com/kiowadriver/unraid-docker in the "Template repositories" field
4. Click on the "Save" button
5. Click back to "Docker" tab and then click on the "Add Container" button
6. Click on the "Template" dropdown menu and select the desired Docker image
7. Click the "Advanced View" toggle on the top right and fill in required fields e.g. volume data, environment variables etc
8. Click on the "Create" button at the bottom of the window to begin pulling down the Docker image
9. Once the image is downloaded you should see it appear in the "Docker Containers" sub-tab

[rmp5s]

Would love to see this added to the wiki! Feel free to add if you'd like 😁 Happy to see your package works well with others!

Which wiki? I'd like to do a nice step by step write-up for people that want to set this up in the future. Only wiki entry I see for it just points here. Where would you like it?

[jef]

https://github.com/jef/streetmerchant/wiki/Installation:-Linux:-Unraid.

Yeah, we can update that wiki to point to here for reference, but also we can include some good information in there as well. Up to you!

[rmp5s]

https://github.com/jef/streetmerchant/wiki/Installation:-Linux:-Unraid.

Yeah, we can update that wiki to point to here for reference, but also we can include some good information in there as well. Up to you!

Got ya.

How about this: I'll do a post that combines @kiowadriver's post and my findings into one post. Then you can just link to that. Should be easy enough for people to find.

Sound good?

[jef]

I think you could update the wiki directly and put whatever you'd like in there! I just figured the link to this discussion could provide some other information for people.

That way you can do a little bit more with that instead of putting up another discussion. Either way works for me though 😀

[rmp5s]

I think you could update the wiki directly and put whatever you'd like in there! I just figured the link to this discussion could provide some other information for people.

That way you can do a little bit more with that instead of putting up another discussion. Either way works for me though 😀

Oh ok, cool! Will do!

[rmp5s]

Hahaha...gotta have EMAIL_USERNAME in there, too!!! 😁

> streetmerchant@ start:production /app
> node build/index.js

ＳＴＲＥＥＴＭＥＲＣＨＡＮＴ
3.3.0

[2:24:44 PM] �[32minfo�[39m :: ℹ selected stores: amazon, bandh, bestbuy, newegg, officedepot, target, walmart
[2:24:44 PM] �[32minfo�[39m :: ℹ selected series: test:series
[2:24:45 PM] �[32minfo�[39m :: ℹ puppeteer config: {
"0": "--disable-dev-shm-usage",
"1": "--no-sandbox",
"2": "--disable-setuid-sandbox",
"3": "--headless",
"4": "--disable-gpu"
}
[2:25:02 PM] �[32minfo�[39m :: 🚀🚨 [amazon] [test:brand (test:series)] test:model :: IN STOCK 🚨🚀
https://www.amazon.com/dp/B083248S3B?tag=randoprojects-20
[2:25:02 PM] �[32minfo�[39m :: ✔ discord message sent
[2:25:02 PM] �[32minfo�[39m :: 🚀🚨 [amazon] [test:brand (test:series)] test:model :: IN STOCK 🚨🚀
https://www.amazon.com/dp/B083248S3B?tag=randoprojects-20
[2:25:02 PM] �[32minfo�[39m :: ✔ discord message sent
[2:25:04 PM] �[32minfo�[39m :: ✔ email sent
[2:25:05 PM] �[32minfo�[39m :: ✔ sms sent
[2:25:10 PM] �[32minfo�[39m :: ℹ [amazon] [test:brand (test:series)] test:model :: IN STOCK, WAITING
[2:25:10 PM] �[32minfo�[39m :: ℹ [amazon] [test:brand (test:series)] test:model :: IN STOCK, WAITING
[2:25:15 PM] �[32minfo�[39m :: ✖ [walmart] [test:brand (test:series)] test:model :: OUT OF STOCK
[2:25:15 PM] �[32minfo�[39m :: ✖ [walmart] [test:brand (test:series)] test:model :: OUT OF STOCK

[rmp5s]

So...you wanna unRAID it up, eh? Well...ya gotta ask yourself..."do I feel lucky?" Well...do ya?

Just kidding. It's actually really easy. Hahaha

Here's how to install the Docker container and customize it to your needs:

First up, to install it, the good @kiowadriver took care of that for you. Here's how to do it:

Prerequisite:  Firstly you need to be running unRAID ver 6.0.0 or later, once installed follow the instructions below...
Navigate to "Docker" tab, then:

1. Enter in a URL of https://github.com/kiowadriver/unraid-docker in the "Template repositories" field
2. Click on the "Save" button
3. Click back to "Docker" tab and then click on the "Add Container" button
4. Click on the "Template" dropdown menu and select the desired Docker image
5. Click the "Advanced View" toggle on the top right and fill in required fields e.g. volume data, environment variables etc
6. Click on the "Create" button at the bottom of the window to begin pulling down the Docker image

Once the image is downloaded you should see it appear in the "Docker" tab.

So, at the bottom of the "Docker" page in unRAID, there is a space to enter the repositories URL...

You'll paste the URL there. Then, you click "add container", then that container from the dropdown at the top.

This will open up a window full of all kinds of stuff, but you really shouldn't need to mess with any of it in order to get it up and running. If you want to customize it, here's how to do it:

In the "normal" way of installing it, you have a .env file that contains all of your settings. If you open that up, on the left of the equals sign, you have the variable, and on the right of the equals sign, between the quotes, you have what we'll call the key...

In that case, a "variable" would be "SLACK_TOKEN" or "SHOW_ONLY_SERIES" while a key would be "amazon,bandh,bestbuy,newegg..."

(Minus the quotes, of course.)

This is all you need to customize your new Docker container. At the bottom, notice there's an "Add another Path, Port..." button...

Click it and you'll get a window like this:

In that window, at the top, click the dropdown and select "variable". Then input your info. In this case, "SHOW_ONLY_SERIES"...

...and click "add". That will give you a new item at the bottom of your "add container" page. Just plop in your "key"...in this case:

And that's it! That's all it takes! I THINK this works with all the streetmerchant variables, but I know it works with Discord, show only series and gmail alerts.

So yea...happy unRAID-ing!!! 😁

[rmp5s]

Hey, you guys been getting tons of timeouts and errors lately? I noticed I haven't gotten any notifications in a while so I pulled up the log for the container and it's 98% timeouts...

I just updated the container...same thing. Still working fine if I run it on my desktop, though...

Anyone else's Docker install go belly up?

[kiowadriver]

My container is up and working fine. I wonder if you were hitting the stores to frequently and got your IP banned momentarily? Try easing up on the timing and using a VPN to test from a different IP.

…

On Thu, Dec 31, 2020 at 11:59 AM rmp5s ***@***.***> wrote: Hey, you guys been getting tons of timeouts and errors lately? I noticed I haven't gotten any notifications in a while so I pulled up the log for the container and it's 98% timeouts... [image: image] <https://user-images.githubusercontent.com/75582315/103420846-a1473080-4b56-11eb-82e1-83d582f89f1a.png> I just updated the container...same thing. Still working fine if I run it on my desktop, though... [image: image] <https://user-images.githubusercontent.com/75582315/103420955-1b77b500-4b57-11eb-9f34-085184e0409d.png> Anyone else's Docker install go belly up? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1428 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDK3HHGRF7B6R33XVPLG6LSXS3WTANCNFSM4U44AO6A> .

[kiowadriver]

Sorry, disregard last. That looks like a bad proxy config to me. If you aren't using a proxy make sure the proxy settings are blank. If you are using a proxy, look at the start message to see if it is configured for one and connects fine.

…

On Thu, Dec 31, 2020 at 1:28 PM Mitchell ***@***.***> wrote: My container is up and working fine. I wonder if you were hitting the stores to frequently and got your IP banned momentarily? Try easing up on the timing and using a VPN to test from a different IP. On Thu, Dec 31, 2020 at 11:59 AM rmp5s ***@***.***> wrote: > Hey, you guys been getting tons of timeouts and errors lately? I noticed > I haven't gotten any notifications in a while so I pulled up the log for > the container and it's 98% timeouts... > > [image: image] > <https://user-images.githubusercontent.com/75582315/103420846-a1473080-4b56-11eb-82e1-83d582f89f1a.png> > > I just updated the container...same thing. Still working fine if I run it > on my desktop, though... > > [image: image] > <https://user-images.githubusercontent.com/75582315/103420955-1b77b500-4b57-11eb-9f34-085184e0409d.png> > > Anyone else's Docker install go belly up? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1428 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACDK3HHGRF7B6R33XVPLG6LSXS3WTANCNFSM4U44AO6A> > . >

[rmp5s]

Sorry, disregard last. That looks like a bad proxy config to me. If you aren't using a proxy make sure the proxy settings are blank. If you are using a proxy, look at the start message to see if it is configured for one and connects fine.
…
On Thu, Dec 31, 2020 at 1:28 PM Mitchell @.> wrote: My container is up and working fine. I wonder if you were hitting the stores to frequently and got your IP banned momentarily? Try easing up on the timing and using a VPN to test from a different IP. On Thu, Dec 31, 2020 at 11:59 AM rmp5s @.> wrote: > Hey, you guys been getting tons of timeouts and errors lately? I noticed > I haven't gotten any notifications in a while so I pulled up the log for > the container and it's 98% timeouts... > > [image: image] > https://user-images.githubusercontent.com/75582315/103420846-a1473080-4b56-11eb-82e1-83d582f89f1a.png > > I just updated the container...same thing. Still working fine if I run it > on my desktop, though... > > [image: image] > https://user-images.githubusercontent.com/75582315/103420955-1b77b500-4b57-11eb-9f34-085184e0409d.png > > Anyone else's Docker install go belly up? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1428 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/ACDK3HHGRF7B6R33XVPLG6LSXS3WTANCNFSM4U44AO6A > . >

Well...there was a port specified on the proxy line so I got rid of that and, same thing...

...it works sometimes. Every once in a while I get a correct "not in stock", but the majority of them are just timeouts. Interesting that yours is working though...I thought maybe the sites changed their layouts or something to break scripts...guess not. lol

[benlikethecolor]

I just setup the container today (thanks rmp5s!) and I'm getting the same timeout errors as you are. I'm wondering if maybe this is occurring because we're querying sites that have broken links? I can't seem to find the certain graphics cards it's looking for on some websites, so I'm wondering if that's the cause. No idea though

[rmp5s]

I just setup the container today (thanks rmp5s!) and I'm getting the same timeout errors as you are. I'm wondering if maybe this is occurring because we're querying sites that have broken links? I can't seem to find the certain graphics cards it's looking for on some websites, so I'm wondering if that's the cause. No idea though

That's what I was thinking as well. I'm still not sure though. Someone more familiar with the deep down inner workings of the script itself may need to look into what's going on. I don't think it has anything to do with Docker, but it may have something to do with the implementation of the script IN Docker...if that makes sense...

tl;dr - No idea what's wrong with it. lol

[benlikethecolor]

I think I've figured it out a bit. I found that when I run this container, in the logs it showed that some of the stores I was running (namely, nvidia and evga) were deprecated. After removing these stores, the number of timeouts decreased drastically, even for stores like amazon and best buy that shouldn't be effected by this. Not sure why this is the case, but I'd suggest people check the top of their logs and see if any of the stores being used are invalid.

[rmp5s]

I think I've figured it out a bit. I found that when I run this container, in the logs it showed that some of the stores I was running (namely, nvidia and evga) were deprecated. After removing these stores, the number of timeouts decreased drastically, even for stores like amazon and best buy that shouldn't be effected by this. Not sure why this is the case, but I'd suggest people check the top of their logs and see if any of the stores being used are invalid.

Hmm. Interesting. I looked at it before and didn't notice anything strange but maybe I missed something. I'll poke my head in there again. Thanks!

[kiowadriver]

Can you share your dotenv? I am not seeing the timeouts. Also, I do think we need to have a link validator on the nightly build ensure all links are valid. Several python scripts are out there that could be leveraged.

…

On Fri, Jan 8, 2021, 11:01 AM Ben Green ***@***.***> wrote: I just setup the container today (thanks rmp5s!) and I'm getting the same timeout errors as you are. I'm wondering if maybe this is occurring because we're querying sites that have broken links? I can't seem to find the certain graphics cards it's looking for on some websites, so I'm wondering if that's the cause. No idea though — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1428 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDK3HBVKHNANUBWJCAOMKLSY427JANCNFSM4U44AO6A> .

[benlikethecolor]

I'm not seeing my dotenv anywhere, if you know where that's generally installed I'd be happy to get you that information (couldn't find it in /home, /root, /app, /tmp, etc).

All I've done to configure my container was setting
"SHOW_ONLY_SERIES" to "3080", and
"STORES" to "amazon,bandh,bestbuy,newegg,officedepot,target,walmart".
No configuration past that, so dotenv shouldn't be that different.

Looking at the logs right now, I can see it timing out on looking at amazon, newegg, bandh, and bestbuy, generally for the same product. My guess is still that it's searching for so many different varients of GPUs that some of those pages don't exist anymore, so it's timing out trying to query that URL, but I have no idea.