Redirekt args with secure link not working

[rebotnix]

i checked your nginx-vod module and it works great with the kaltura services.

i do not know if this is an issue to report, but when we enable the nginx secure link function:

secure_link $arg_checksum,$arg_timestamp;

master looks like:
.../master.m3u8?timestamp=DefinedInMS&checksum=md5checksum, this is going thru the validation process of the internal secure link function of nginx. After this the kaltura VOD-module redirects to:
./index-v1-a1.m3u8

"GET ....index-v1-a1.m3u8".....gives a 501 error,

referer was from "/master.m3u8?timestamp=1430348566&md5checksum=PG4LipwVIkqCKLmpjKFTHQ"

As you can see, it works for the kaltura-vod-module generated master.m3u8 for hls streaming, but the redirected single index.m3u8 playlist cuts out all arguments that where transferred and validated with the MD5 checksum of the first request.

Is there a way that you please fix that, maybe with an flag to enable and pass thru the arguments in the generated index m3u8 files?

i tested also the dash,hds and mss, same result here.

thanks.

[erankor]

I don't think passing the query arguments to the manifest URLs will solve your problem, since afaik it is not possible to sign part of the URL with nginx's secure link module out of the box (it is probably possible with a custom module that adds a uri_path variable to nginx, or something like that). In other words, the token passed on the manifest request will not work for the video segments.

As I understand, what you're trying to do is have secure video URLs, I can suggest a different solution using 2 other Kaltura modules:

1. https://github.com/kaltura/nginx-secure-token-module
2. https://github.com/kaltura/nginx-akamai-token-validate-module
   The purpose of the first module is to embed tokens inside video manifest responses (hls-m3u8, hds-f4m, dash-mpd). It is currently capable of generating 2 types of tokens - Akamai V2 tokens and Amazon CloudFront tokens.
   Since you are probably aren't working with these CDNs and you're serving the content directly from nginx, you can use the second module I listed to validate these tokens (replacing nginx's secure link)
   So the flow would be:
3. generate an Akamai V2 token to secure the manifest request
4. request reaches the nginx server and token is validated by nginx-akamai-token-validate
5. if the validation passes, nginx-vod-module generates the manifest
6. nginx-secure-token module processes the response and embeds tokens on the URLs contained in the manifest
7. the tokens that were embedded on the segment URLs are validated by nginx-akamai-token-validate

Let me if you need more info on how to configure that

[rebotnix]

great!

You were right about the chunks, which i forgot as i wrote the issue-ticket. I have not seem that akamai token and cloud front module exits, thanks a lot for showing this to me. A little bit of help how i can generate a validate link would be very helpful. I will try to understand it from the code, but when you can show me a command or shell script how to generate the secure token, that would be helpful.

Here is maybe an interesting information why i want to have an alternate secure link and not use only one CDN provider:
I would like to build a CDN balancer , not a classic server DNS round robin balancer.

I want to use akamai server and cloud front, but i also want to balance them with other provider/servers. The balancer handles automatically the first 1-100 connection from our local server (cheaper then CDN streaming) and when the number of users grow up, i switch and turn on the CDN balancer.

Maybe you ask why i don't use one CDN? The reality is that CDN providers uses different caching nodes where these nodes were installed on different peerings locations. The GEO locations from these nodes were sometimes not the best peerings to the location to our clients peerings.

Cause i do not get a QOS (quality of server) feedback from the caching nodes when a client from us gets in a buffer under situation, i add a own QOS event manager in our players which reports the IP from the caching node to our CDN balancer. The more events were are receiving and collecting from our clients, the more is the transparency of the whole network from the used CDN.

I was very surprised to see how overloaded some peerings from big CDN provider were on different times and from the network from the client provider to the caching-node. This seems to be normal, cause internet always works in a best afford modell.

For this reasons i have to use an global secure link function to protect our videos thru our own custom build multiple CDN´s.

That cloudfront and akamai token exits, helped me a lot.

[erankor]

Here is some PHP code that generates these tokens in the Kaltura API server:
https://github.com/kaltura/server/blob/Jupiter-10.10.0/alpha/apps/kaltura/lib/storage/urlTokenizers/kCloudFrontUrlTokenizer.php#L61
https://github.com/kaltura/server/blob/Jupiter-10.10.0/alpha/apps/kaltura/lib/storage/urlTokenizers/kAkamaiSecureHDUrlTokenizer.php#L204

There's also C code that does the same in nginx-secure-token-module but it's easier to read in PHP, hope it helps.

Btw, didn't understand if that is what you already planned to do, but note that you have use tokens that correspond to each CDN configuration in order to enable the CDN to cache the content. For example, if you use nginx's secure link token with Akamai, the CDN nodes will not be able to validate the token and all the requests will reach your servers. This means you will need to configure a separate location in nginx.conf for each combination of (protocol, CDN). Non-tokenized configurations can be shared between different CDNs.

[rebotnix]

hi erankor,

i use the akamai token validator and was, thanks to your help, able to install a secure token module and replace the original secure link from nginx. i study some other stuff and i was impressed how much work you did for the nginx server which is now my favorite one.

if you ever need a place to sleep in germany or having a beer, let me know :)

may the source with you...

[rebotnix]

sorry for coming back to this again, but i still have a problem to get thru to the single manifest and chunks using the akamai token validator plugin.

Please take look what i did:

1. compile nginx with vod and akamai token validator
2. create a demo location in my server.conf:

local hls playback

     location /hls/ {
        alias /opt/webcontent/;
        akamai_token_validate on;
        akamai_token_validate_key 000000000000; # only for testing
        ...

3. test the link to /hls/myvideo.mp4/master.m3u8 - gave me a 403 forbidden, which is right cause token is switched on
4. generate a token, like this:
   hdnea=st=1430611557exp=1430617557acl=/hls/s400k.mp4/master.m3u8~hmac=45f4e52072a22f511c57f9c88bc073fed97a95db443c14b1f3eaabc12c84e04f
5. request the hls master.m3u8 like this (the hdnea value has underscopes of course).
   http://myserver:8080/hls/s400k.mp4/master.m3u8?**hdnea**=st=1430611557~exp=1430617557~acl=/hls/s400k.mp4/master.m3u8~hmac=45f4e52072a22f511c57f9c88bc073fed97a95db443c14b1f3eaabc12c84e04f
6. with this uri i´m getting access to the master.m3u8, so the token seems to be working, but then the redirected manifest in the master.m3u8, which is "**index-v1-a1.m3u8" has no token information, btw. i saw a 403 forbidden in the error.log file.

The master.m3u8 looks like this (requested with token)
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1650611,RESOLUTION=720x406,CODECS="avc1.4d401e,mp4a.40.2"
http://myserver:8080/hls/s400k.mp4/index-v1-a1.m3u8

It seems that i´m now in the same situation as with the secure link plugin from nginx.

Can you please tell me what i´m doing wrong?

Thanks a lot.

[rebotnix]

I red your post again and i think i have to include the secure module, not only the validation. I need a break and will try tomorrow again :)

[erankor]

yup, you need to use nginx-secure-token-module to embed tokens in the response, here is a sample config you can use:
secure_token akamai;
secure_token_akamai_key 000000000000;
secure_token_akamai_acl "$baseuri*";
secure_token_types application/vnd.apple.mpegurl video/f4m application/dash+xml text/xml;
secure_token_expires_time 100d;
secure_token_cookie_token_expires_time 0;
secure_token_query_token_expires_time 1h;

in our config we set all the parameters in the 'server' level in nginx.conf except 'secure_token akamai' which we enable only in specific locations. We have some locations that have Akamai tokens, some CF tokens, and some without tokens at all, so it's convenient to avoid copying the parameters between all akamai tokenized locations, for example.

[rebotnix]

YES!

it works perfectly now....I have now understood the in and outs of the token modules. I will now try to run elastic search, logstash and kibana on top. So amazing to see whats possible with the kaltura stuff. Thanks for all the help!

[rbu]

@erankor, you stated before:

I don't think passing the query arguments to the manifest URLs will solve your problem, since afaik it is not possible to sign part of the URL with nginx's secure link module out of the box [...] In other words, the token passed on the manifest request will not work for the video segments.

There's actually a neat trick in the nginx docs to only sign and validate the base of the URI, at http://nginx.org/en/docs/http/ngx_http_hls_module.html#hls_forward_args

Given a config like so:

http {
    ...

    map $uri $hls_uri {
        ~^(?<base_uri>.*).m3u8$ $base_uri;
        ~^(?<base_uri>.*).ts$   $base_uri;
        default                 $uri;
    }

    server {
        ...

        location /hls {
            secure_link $arg_md5,$arg_expires;
            secure_link_md5 "$secure_link_expires$hls_uri$remote_addr secret";
...
        }
    }
}

You can now generate a link secret using without specifying .ts/.m3u8 at the end and the nginx module would only have to reproduce the arguments to the m3u8 file in the playlist (so they are reflected to nginx for accessing the .ts files).
If ts and m3u8 files share no base name, the secret could just be generated over the directory name.

[erankor]

Nice, thanks for sharing.
And yes, the regular expression would need to be changed to match up to the last /, something like (.*)/[^/]+

[ghost]

erankor, sorry very much, but where need to change regular expr. (.*)/[^/]+ ?
like this? :)

map $uri $hls_uri {
        ~^(.*)/[^/]+.m3u8$ $base_uri;
        ~^(.*)/[^/]+.ts$   $base_uri;
        default                 $uri;
}

[erankor]

like this:

map $uri $hls_uri {
        ~^(?<base_uri>.*)/[^/]+$   $base_uri;
        default                    $uri;
}

[ghost]

Don't write arguments to m3u8 :(

#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-1-v1-a1.ts
#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-2-v1-a1.ts
#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-3-v1-a1.ts
#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-4-v1-a1.ts
#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-5-v1-a1.ts

[erankor]

I don't understand what you mean

[ghost]

I enable secure link mod.

m3u8 file need to be like
#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-1-v1-a1.ts?md5=key&expires=time

right? but generated m3u8 file is

#EXTINF:2.000,
http://v1.localhost.lc/1/hls/test.mp4/seg-1-v1-a1.ts

[erankor]

I believe you can configure it so that the token will be on the path of the URL instead of the query string, and then it will propagate to the segments. But anyway, as previously said, we are working with a CDN so we don't use nginx's secure link.

[kohaku1907]

sorry for bother you, but i have a problem when using akamai token validator plugin.
Please take look what i did:
Here my config

secure_token_akamai $token {
        key 1234;
        acl "$secure_token_baseuri*";
}

location /hls {
    alias /mnt/gcs-media/vod;

    vod hls;
    ......
    secure_token $token;
    secure_token_types application/vnd.apple.mpegurl video/f4m application/dash+xml text/xml;
    secure_token_expires_time 100d;
    secure_token_query_token_expires_time 1h;
    akamai_token_validate on;
    akamai_token_validate_key 1234;

}

test the link to /hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8 - gave me a 403 forbidden
generate a token, like this:
My php code when use this class https://github.com/kaltura/server/blob/Lynx-12.9.0/alpha/apps/kaltura/lib/storage/urlTokenizers/kAkamaiSecureHDUrlTokenizer.php

require_once ('kAkamaiSecureHDUrlTokenizer.php');
$kAkamaiSecureHDUrlTokenizer = new kAkamaiSecureHDUrlTokenizer();
$kAkamaiSecureHDUrlTokenizer->setWindow(300);
$kAkamaiSecureHDUrlTokenizer->setKey('1234');
$url = $kAkamaiSecureHDUrlTokenizer->tokenizeSingleUrl('/hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8');
echo $url;

Then i receive this
/hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8?=st=1484109706exp=1484110006acl=/hls/sample/Football_Made_in_Brazil_Trailer.mp4~hmac=fdd88464bf66763c235bf9f99649b935261af47dc64413b871aace773f6f27a2

request the hls master.m3u8 like this
http://myserver:8888/hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8?=st=1484109706~exp=1484110006~acl=/hls/sample/Football_Made_in_Brazil_Trailer.mp4~hmac=fdd88464bf66763c235bf9f99649b935261af47dc64413b871aace773f6f27a2

i saw a 403 forbidden in the access.log file
Can you please tell me what i´m doing wrong?

Thanks a lot.

[erankor]

Add to the script:

$kAkamaiSecureHDUrlTokenizer->setAclPostfix('*');

What happens is that the /master.m3u8 part gets stripped, and since the tokenizer is not configured to add an asterisk, you get an ACL that doesn't match the URL.

[kohaku1907]

It 's still not worked.
It generated beblow
/hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8?=st=1484113828~exp=1484114128~acl=/hls/sample/Football_Made_in_Brazil_Trailer.mp4*~hmac=cc6d78965047beaee546ac9f6e05ec5e512f41fada52d8c3e42a14147333760e

In log format : i add variable $secure_token_baseuri to detect baseuri
In access.log :
[11/Jan/2017:13:03:08 +0700] "GET /hls/sample/Football_Made_in_Brazil_Trailer.mp4/master.m3u8?=st=1484113828~exp=1484114128~acl=/hls/sample/Football_Made_in_Brazil_Trailer.mp4*~hmac=cc6d78965047beaee546ac9f6e05ec5e512f41fada52d8c3e42a14147333760e HTTP/1.1" 403 199 "http://bitmovin.com/hls-mpeg-dash-test-player/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" "-" "-"/hls/sample/Football_Made_in_Brazil_Trailer.mp4/
Thanks a lot

[kohaku1907]

It worked
I forgot add this :
$kAkamaiSecureHDUrlTokenizer->setParamName('__hdnea__');

Thanks a lot

[kohaku1907]

sorry for bother you again. I try to turn on validate ip_address of client ( diffirent url per IP )
secure_token_akamai $token {
key 1234;
acl "$secure_token_baseuri*";
ip_address $remote_addr;
}

But it 's not work . I can play anylinks generated by PHP script above with any ip .

[erankor]

IP validation is not implemented, see the last comment here - kaltura/nginx-akamai-token-validate-module#5 (comment)

[pakitv]

@erankor
please let me know [email protected]
vbr

[erankor]

@pakitv, I don't follow, let you know what?