Skip to content

Commit ffe2f95

Browse files
slloyd-srcslloyd-src
committed
- Removed unused properties
- Moved IAM, group & bear token details to detail.md
1 parent 95b7e6c commit ffe2f95

File tree

7 files changed

+150
-111
lines changed

7 files changed

+150
-111
lines changed

‎README.md

+38-85
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
caomdev
21
=======
32

43
Fire up a local development CAOM db instance. This will only create a local instance of the database with the REST API that the https://github.com/uksrc/emerlin2caom can talk to. There is some [background detail](detail.md) of the services that need to be set up.
@@ -21,11 +20,11 @@ Several components that have been developed by OpenCADC are used in this deploym
2120
- [Torkeep - API access to database](https://github.com/opencadc/caom2db/tree/main/torkeep)
2221

2322

24-
## Using
23+
## Deploying
2524
1. Install self-signed certificates
26-
- RootCA.crt in browser and command line
25+
- RootCA.crt in browser and command line. See [Ubuntu example](https://ubuntu.com/server/docs/install-a-root-ca-certificate-in-the-trust-store) and [Firefox example](https://docs.vmware.com/en/VMware-Adapter-for-SAP-Landscape-Management/2.1.0/Installation-and-Administration-Guide-for-VLA-Administrators/GUID-0CED691F-79D3-43A4-B90D-CD97650C13A0.html) for linux reference.
2726
- Domain for this is https://src-data-repo.co.uk
28-
- Can be changed but you'll have to create your own, repeat 1 above, update the nginx.conf and replace the rootCA.crt in each service's config folder (so they can trust each other).
27+
- Can be changed but you'll have to create your own certificate & root authority, repeat 1 above, update the nginx.conf and replace the rootCA.crt in each service's config folder (so they can trust each other).
2928
<br>
3029
<br>
3130
2. Install Docker (& docker-compose)
@@ -36,6 +35,8 @@ Several components that have been developed by OpenCADC are used in this deploym
3635
3. Clone repository
3736
https://github.com/uksrc/caomdev
3837
<br>
38+
4. Adjust settings
39+
See [detail.md](detail.md) for information reagrding identity managment, permissions groups & bearer tokens for requests.
3940
<br>
4041
4. Start the postgres db (done separately)<br>
4142

@@ -58,47 +59,45 @@ Several components that have been developed by OpenCADC are used in this deploym
5859
docker-compose -f docker-compose-dbase.yml down
5960
```
6061

61-
### Testing
62+
## Testing
6263

6364
Each component has a couple of standard 'status' APIs (returns XML):
6465

65-
Get the status of the component
66-
https://<em>\<domain\></em>/<em>\<component\></em>/availability
66+
#### Get the status of the component(s)
67+
>https://<em>\<domain\></em>/<em>\<component\></em>/availability
6768
68-
Get a list of available APIs <br>
69-
https://<em>\<domain\></em>/<em>\<component\></em>/capabilities
69+
#### Get a list of available APIs
70+
>https://<em>\<domain\></em>/<em>\<component\></em>/capabilities
7071
71-
Can be called like this from the command line (or from the browser)
72+
Can be called like this from the command line (or use the URL in a browser)
7273
```
7374
curl -k https://src-data-repo.co.uk/torkeep/availability
7475
```
75-
These should work for <em>reg</em>, <em>baldur</em> or <em>torkeep</em>.
76+
These should work for the <em>reg</em>, <em>baldur</em> or <em>torkeep</em> components.
7677

7778

78-
List the **registry contents**
79+
#### List the registry contents
7980
```
8081
curl -k https://src-data-repo.co.uk/reg/resource-caps
81-
```
82-
Shoud return a list of services that were defined in <em>./config/reg/reg-resource-caps.properties</em>
83-
```
82+
83+
# Should return a list of services that were defined in <em>./config/reg/reg-resource-caps.properties
84+
8485
#First, global services:
8586
ivo://skao.int/reg = https://src-data-repo.co.uk/reg/capabilities
8687
ivo://skao.int/gms = https://ska-gms.stfc.ac.uk/gms/capabilities
8788
ivo://skao.int/baldur = https://src-data-repo.co.uk/baldur/capabilities
88-
89-
...
9089
```
9190

92-
**Group permissions** as defined in <em>./config/baldur/baldur.properties</em>
91+
92+
#### Group permissions as defined in <em>./config/baldur/baldur.properties</em>
9393
Extra info here - https://github.com/opencadc/storage-inventory/tree/main/baldur
9494

95-
curl https://<em>\<domain\></em>/baldur/perms?op=<em>grantType</em>\&ID=<em>identifier</em>
96-
```
97-
curl https://src-data-repo.co.uk/baldur/perms?op=read\&ID=caom:EMERLIN/
95+
>curl https://<em>\<domain\></em>/baldur/perms?op=<em>grantType</em>\&ID=<em>identifier</em>
9896
```
97+
> curl https://src-data-repo.co.uk/baldur/perms?op=read\&ID=caom:EMERLIN/
98+
99+
# Should return (if found), details of the group
99100
100-
Should return (if found), details of the group
101-
```
102101
<?xml version="1.0" encoding="UTF-8"?>
103102
<grant type="ReadGrant">
104103
<assetID>caom:EMERLIN/</assetID>
@@ -109,84 +108,38 @@ Should return (if found), details of the group
109108
# assetID pattern needs to conform to caom:{collection}/{observationID}
110109
```
111110

112-
⚠️ **Warning:** Be cautious of the pattern used to match, see <em>baldur.properties</em>' **EMERLIN.pattern** for the regular expression used to match the search term.
113-
111+
⚠️ **Warning:** Be cautious of the pattern used to match, see [baldur.properties](config/baldur/baldur.properties)' <em>\<entry name\></em>.pattern for the regular expression used to match the search term.
114112

113+
<br>
115114

116-
**Database submission & retrieval** (torkeep service)
115+
#### Database submission & retrieval (torkeep service)
117116
https://src-data-repo.co.uk/torkeep/ in a browser for a detailed list of available APIs in a more readable fashion than calling <em>../torkeep/capabilities</em>
118117

119-
A call to observations returns the available groups
118+
**Note** A bearer token is required for write and delete requests (shown as "SKA_TOKEN" below), see details.md for user account & bear token information.
120119
```
121120
> curl https://src-data-repo.co.uk/torkeep/observations
122-
test
123-
EMERLIN
124-
```
125-
126-
127-
### Bearer Token for API calls using group permissions
128-
https://confluence.skatelescope.org/display/SRCSC/RED-10+Using+oidc-agent+to+authenticate+to+OpenCADC+services
129-
```
130-
curl -SsL --header "authorization: bearer $SKA_TOKEN" https://src-data-repo.co.uk/torkeep/observations
131-
132-
#### Franz Kirsten's Repo (for reference)
133-
https://gitlab.com/users/fkirsten/projects
134-
135121
136-
### Certs and command line custom domain
137-
As we're currently using a custom domain name, we need to allow the browser to authenticate our own tickets.
138-
https://docs.vmware.com/en/VMware-Adapter-for-SAP-Landscape-Management/2.1.0/Installation-and-Administration-Guide-for-VLA-Administrators/GUID-0CED691F-79D3-43A4-B90D-CD97650C13A0.html
122+
# Should return the list of groups (collections) defined in baldur.properties
139123
140-
Also, we can do this for command line calls such as curl requests
141-
142-
143-
User needs to be a member of a permissions group via https://ska-iam.stfc.ac.uk/login.
144-
Currently set to ivo://skao.int/gms?prototyping-groups/mini-src/platform-users but can be changed in the baldur.properties file.
145-
```
146-
cp rootCa and tls.crt to /usr/local/share/ca-certificates/
147-
148-
sudo update-ca-certificates
149-
```
150-
151-
The domain will need to be added to the "hosts" file
152-
153-
Linux:
154-
```
155-
>nano /etc/hosts
156-
```
157-
There will be an entry(entries) such as:
158-
```
159-
127.0.0.1 localhost
160-
```
161-
Add another:
162-
```
163-
127.0.0.1 src-data-repo.co.uk
164-
```
165-
Should hopefully work immediately.
124+
test
125+
EMERLIN
166126
167-
I’ve never done it on Mac so I just googled this https://www.nexcess.net/help/how-to-find-the-hosts-file-on-my-mac/
127+
# Inject some data
128+
> curl -v --header "Content-Type: text/xml" --header "authorization: bearer $SKA_TOKEN" -T test_data.xml https://src-data-repo.co.uk/torkeep/observations/EMERLIN/TS8004_C_001_20190801_avg.ms
168129
169-
On Windows it’s <em>C:\Windows\System32\drivers\etc\hosts</em>
130+
# Check what's been stored for a named collection
131+
> curl https://src-data-repo.co.uk/torkeep/observations/EMERLIN
132+
EMERLIN minimal-observation 2024-08-22T10:56:37.252 md5:f1a40291ce1dd85623a43d0c2b3b3758
133+
EMERLIN TS8004_C_001_20190801_avg.ms 2024-08-22T11:05:31.771 md5:260c09954bcb7494e0ca8255aa3ec743
170134
135+
# Delete an entry
136+
> curl -X DELETE --header "authorization: bearer $SKA_TOKEN" -T test_data.xml https://src-data-repo.co.uk/torkeep/observations/EMERLIN/TS8004_C_001_20190801_avg.ms
171137
172-
## Usage
173138
```
174-
### Don't forget to get a new token (1 hour expiration)
175-
export SKA_TOKEN=$(oidc-token example-client)
176-
177-
### Make sure observationID and collection are the same in the file as used in the curl request.
178139

179-
### PUT a new entry
180-
curl -v --header "Content-Type: text/xml" --header "authorization: bearer $SKA_TOKEN" -T test_data.xml https://src-data-repo.co.uk/torkeep/observations/EMERLIN/TS8004_C_001_20190801_avg.ms
140+
<br>
181141

182-
### Read the observations under a named collection, read operations shouldn't need the SKA_TOKEN whilst set to anon = true in the baldur.properties.
183-
curl -X GET --header 'Accept: text/tab-separated-values' 'https://src-data-repo.co.uk/torkeep/observations/EMERLIN'
184142

185-
### Delete a named entry
186-
curl -X DELETE --header "authorization: bearer $SKA_TOKEN" https://src-data-repo.co.uk/torkeep/observations/EMERLIN/TS8004_C_001_20190801_avg.ms
187-
```
188143

189-
### Notes
190-
<em>start-services.sh</em> has to be used to populate HOST_IP required by the docker-compose file, rather than calling <em>docker-compose up</em> directly. This value is used by the containers to allow access to the custom domain and will allow access via the reverse proxy. It's basically the IP address of the host machine from the perspective of the docker containers.
191144

192145

‎config/baldur/baldur.properties

+2-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ org.opencadc.baldur.allowAnon = true
1212
org.opencadc.baldur.entry = test
1313
test.pattern = ^cadc:TEST/.*
1414
test.anon = true
15-
#test.readOnlyGroups = ivo://skao.int/gms?prototyping-groups/mini-src/platform-users
16-
#test.readWriteGroups = ivo://skao.int/gms?prototyping-groups/mini-src/platform-users
15+
test.readOnlyGroups = ivo://skao.int/gms?prototyping-groups/mini-src/platform-users
16+
test.readWriteGroups = ivo://skao.int/gms?prototyping-groups/mini-src/platform-users
1717

1818
org.opencadc.baldur.entry = EMERLIN
1919
EMERLIN.pattern = ^caom:EMERLIN/.*

‎config/reg/catalina.properties

-2
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,3 @@ tomcat.connector.proxyPort=443
99

1010
## enable no-op authenticator plugin
1111
ca.nrc.cadc.auth.Authenticator=ca.nrc.cadc.auth.NoOpAuthenticator
12-
13-
#ca.nrc.cadc.auth.IdentityManager=org.opencadc.auth.StandardIdentityManager

‎config/reg/reg-resource-caps.properties

+1-11
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,6 @@
1-
#First, global services:
1+
#Global services:
22

33
ivo://skao.int/reg = https://src-data-repo.co.uk/reg/capabilities
4-
#ivo://skao.int/gms = https://ska-iam.stfc.ac.uk/gms/capabilities
54
ivo://skao.int/gms = https://ska-gms.stfc.ac.uk/gms/capabilities
65
ivo://skao.int/baldur = https://spsrc27.iaa.csic.es/baldur/capabilities
76
ivo://mansrc.int/baldur = https://src-data-repo.co.uk/baldur/capabilities
8-
9-
#The `reg` self-registration is not actually used by anything, but it claims that resourceID for that service. The `gms` URL is probably wrong - the example is just to show it's related to IAM server but the value will come from purple team.
10-
11-
12-
13-
#Then SRC node services:
14-
15-
#ivo://swesrc.chalmers.se/luskan = https://se.swesrc.chalmers.se/luskan/capabilities
16-
#ivo://swesrc.chalmers.se/minoc = https://se.swesrc.chalmers.se/minoc/capabilities

‎config/torkeep/catalina.properties

-3
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,5 @@ tomcat.connector.scheme=https
1616
tomcat.connector.proxyName=src-data-repo.co.uk
1717
tomcat.connector.proxyPort=443
1818

19-
## enable no-op authenticator plugin -- obsolete because of the useage of the OIDC IdentityManager below
20-
#ca.nrc.cadc.auth.Authenticator=ca.nrc.cadc.auth.NoOpAuthenticator
21-
2219
## enable OIDC IdentityManager
2320
ca.nrc.cadc.auth.IdentityManager=org.opencadc.auth.StandardIdentityManager

‎config/torkeep/torkeep.properties

-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
org.opencadc.torkeep.collection = test
22
test.basePublisherID = ivo://skao.int/
33
test.computeMetadata = false
4-
#test.proposalGroup = false
54

65
org.opencadc.torkeep.collection = EMERLIN
76
EMERLIN.basePublisherID = ivo://skao.int/

0 commit comments

Comments
 (0)