libertyEventDrivenSurvey

libertyEventDrivenSurvey is an example event-driven survey application demonstrating Liberty InstantOn, CloudEvents, KNative, and MicroProfile Reactive Messaging 3.

The way it works is that users scan a QR code presented by the person running the survey and users type in a location (e.g. New York). This is submitted to a microservice that can scale from zero using KNative Serving and quickly using Liberty InstantOn. This web service publishes the location to a Kafka topic. Another microservice that can scale from zero using KNative Eventing and the KNative Kafka Broker and quickly using Liberty InstantOn then receives this event and geocodes the location to a latitude and longitude through a Geoapify or Google Maps API. Finally, this pin works its way back through another Kafka topic and WebSockets back to the map that the person running the survey is showing.

Screeenshots and Architecture for Geocoding

Installing the application

1. Perform the Development steps.
2. Perform the Deploy to OpenShift >= 4.13 steps.

Development

1. If using podman machine:
   1. Set your connection to the root connection:

      podman system connection default podman-machine-default-root
      

   2. If the machine has SELinux virt_sandbox_use_netlink disabled (i.e. the following returns off):

      podman machine ssh "getsebool virt_sandbox_use_netlink"
      

      Then, enable it:

      podman machine ssh "sudo setsebool virt_sandbox_use_netlink 1"
      

      Note that this must be done after every time the podman machine restarts.
2. Build:

   mvn clean deploy
   

Deploy to OpenShift >= 4.13

Pre-requisities

1. Install Kafka; for example, the Red Hat Streams for Apache Kafka operator
   1. Create project amq-streams-kafka
   2. OperatorHub } AMQ Streams } A specific namespace = amq-streams-kafka
   3. Kafka } Create Instance } my-cluster } Use all default options } Create
   4. Wait for Condition: Ready
      1. If you get a warning about "inter.broker.protocol.version", apply the known workaround
   5. Kafka Topic } Create KafkaTopic } locationtopic } Create
   6. Kafka Topic } Create KafkaTopic } geocodetopic } Create
2. Install KNative; for example, the Red Hat OpenShift Serverless operator
   1. Install the kn command line utility
      1. Alternatively, install the latest version of kn and the kafka plugin
         1. macOS:

            brew install knative/client/kn knative-sandbox/kn-plugins/source-kafka
            

   2. Install KNative Serving
      1. Operators } Installed Operators
      2. Project = knative-serving
      3. Red Hat OpenShift Serverless } Knative Serving
      4. Create KnativeServing
      5. Click Create
      6. Wait for the Ready Condition in Status
   3. Install KNative Eventing
      1. Operators } Installed Operators
      2. Project = knative-eventing
      3. Red Hat OpenShift Serverless } Knative Eventing
      4. Create KnativeEventing
      5. Click Create
      6. Wait for the Ready Condition in Status
   4. Install the KNativeKafka broker
      1. Knative Kafka } Create KnativeKafka
      2. channel } enabled; bootstrapServers } my-cluster-kafka-bootstrap.amq-streams-kafka.svc:9092
      3. source } enabled
      4. broker } enabled; defaultConfig } bootstrapServers } my-cluster-kafka-bootstrap.amq-streams-kafka.svc:9092
      5. sink } enabled
      6. Click Create
      7. Wait for the Ready Condition in Status
3. Change directory to this cloned repository:

   cd .../libertyEventDrivenSurvey
   

4. Check the current project is some test project name:

   oc project
   

   1. If not, create and switch to some test project:

      oc new-project libertysurvey
      

5. Ensure the internal OpenShift registry is available:

   oc patch configs.imageregistry.operator.openshift.io/cluster --patch "{\"spec\":{\"defaultRoute\":true}}" --type=merge
   

6. Choose your geocoding provider:
   1. Get a Geoapify key (default)
      1. Free commercial usage:

         When you stay within the Free pricing plan quota, you can use the Maps API for free, even for a commercial app.

      2. Attribution policy

         Geoapify attribution is mandatory when using Free subscription plan.

   2. Get a Google Maps API key (simple usage should fit within the free tier)
      1. In general, it's recommended to use a restricted API key in case it is stolen. If you would like to do this, note that the same API key is used both by the JavaScript frontend in the browser and one of the services running in Kubernetes, so both would need to be allowed (e.g. by IP, etc.).
      2. After creating the API key, go to Enabled APIs & services, click ENABLE APIS AND SERVICES, and make sure that Maps JavaScript API and Places API are enabled.
7. Create a service account for InstantOn:

   oc create serviceaccount instanton-sa
   

8. Create an InstantOn SecurityContextConstraints:

   oc apply -f lib/instantonscc.yaml
   

9. Associate the InstantOn SecurityContextConstraints with the service account:

   oc adm policy add-scc-to-user cap-cr-scc -z instanton-sa
   

10. To use InstantOn, we need to modify KNative Serving configuration which must be done through the operator:
    1. Operators } Installed Operators
    2. Project = knative-serving
    3. Red Hat OpenShift Serverless } Knative Serving
    4. Click knative-serving
    5. Click YAML
    6. Under spec, add:

         config:
           features:
             kubernetes.containerspec-addcapabilities: enabled
             kubernetes.podspec-securitycontext: enabled
       

    7. Click Save

Kafka Security

If you want to enable Kafka Security (e.g. SASL), then you will need to follow the relevant steps in Kafka connector security configuration and change the relevant YAML configurations below. This may involve, for example, adding a keystore to some of the containers (either during the build phase or by mounting a directory).

Deploy surveyInputService

1. Push surveyInputService to the registry:

   REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
   echo "Registry host: ${REGISTRY}"
   printf "Does it look good (yes=ENTER, no=Ctrl^C)? "
   read trash
   podman login --tls-verify=false -u $(oc whoami | sed 's/://g') -p $(oc whoami -t) ${REGISTRY}
   podman tag localhost/surveyinputservice $REGISTRY/libertysurvey/surveyinputservice
   podman push --tls-verify=false $REGISTRY/libertysurvey/surveyinputservice
   

2. Create a KNative Service for surveyInputService (if needed, replace mp.messaging.connector.liberty-kafka.bootstrap.servers with the AMQ Streams Kafka Cluster bootstrap address):

   oc apply -f lib/example_surveyinputservice.yaml
   

3. Query until READY is True:

   kn service list surveyinputservice
   

4. Open your browser to the URL from the kn service list output above and click on Location Survey.
5. Double check logs look good:

   oc exec -it $(oc get pod -o name | grep surveyinputservice) -c surveyinputservice -- cat /logs/messages.log
   

6. Note that scale-down-delay does not apply to the initial pod creation so the pod will be terminated about 30 seconds after it's initially created. Once a real user request is made to this application, then scale-down-delay will apply. Therefore, if you want to tail the logs of the pod, first wait for the initial pod to terminate and then make a request to the application and then you can tail the pod logs.

Deploy surveyAdminService

1. Push surveyAdminService to the registry:

   REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
   echo "Registry host: ${REGISTRY}"
   printf "Does it look good (yes=ENTER, no=Ctrl^C)? "
   read trash
   podman login --tls-verify=false -u $(oc whoami | sed 's/://g') -p $(oc whoami -t) ${REGISTRY}
   podman tag localhost/surveyadminservice $REGISTRY/libertysurvey/surveyadminservice
   podman push --tls-verify=false $REGISTRY/libertysurvey/surveyadminservice
   

2. Copy lib/example_surveyadminservice.yaml.template into lib/example_surveyadminservice.yaml, and then:
   1. If using Google as the map provider, add a GOOGLE_API_KEY env entry with your Google Maps API key
   2. Replace INSERT_URL with the URL from the serviceInputService above appended with location.html
   3. If needed, replace SURVEY_LATITUDE and SURVEY_LONGITUDE (defaults to Las Vegas, NV, USA)
   4. If needed, replace mp.messaging.connector.liberty-kafka.bootstrap.servers with the AMQ Streams Kafka Cluster bootstrap address
   5. Run:

      oc apply -f lib/example_surveyadminservice.yaml
      

3. Query until READY is True:

   kn service list surveyadminservice
   

4. Open your browser to the URL from the kn service list output above and click on Start New Geolocation Survey.
5. Double check logs look good:

   oc exec -it $(oc get pod -o name | grep surveyadminservice) -c surveyadminservice -- cat /logs/messages.log
   

6. Create a KNative Eventing KafkaSource for surveyAdminService (if needed, replace bootstrapServers with the AMQ Streams Kafka Cluster bootstrap address):

   oc apply -f lib/example_surveyadminkafkasource.yaml
   

7. Query until OK is ++ for all lines:

   kn source kafka describe geocodetopicsource
   

8. Note that scale-down-delay does not apply to the initial pod creation so the pod will be terminated about 30 seconds after it's initially created. Once a real user request is made to this application, then scale-down-delay will apply. Therefore, if you want to tail the logs of the pod, first wait for the initial pod to terminate and then make a request to the application and then you can tail the pod logs.

Deploy surveyGeocoderService

1. Push surveyGeocoderService to the registry:

   REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
   echo "Registry host: ${REGISTRY}"
   printf "Does it look good (yes=ENTER, no=Ctrl^C)? "
   read trash
   podman login --tls-verify=false -u $(oc whoami | sed 's/://g') -p $(oc whoami -t) ${REGISTRY}
   podman tag localhost/surveygeocoderservice $REGISTRY/libertysurvey/surveygeocoderservice
   podman push --tls-verify=false $REGISTRY/libertysurvey/surveygeocoderservice
   

2. Copy lib/example_surveygeocoderservice.yaml.template into lib/example_surveygeocoderservice.yaml, and then:
   1. If using Geoapify as the geocoding provider (default), add a GEOAPIFY_API_KEY env entry with your Geoapify API key
   2. If using Google as the geocoding provider, add a GOOGLE_API_KEY env entry with your Google Maps API key
   3. If needed, replace mp.messaging.connector.liberty-kafka.bootstrap.servers with the AMQ Streams Kafka Cluster bootstrap address
   4. Run:

      oc apply -f lib/example_surveygeocoderservice.yaml
      

3. Query until READY is True:

   kn service list surveygeocoderservice
   

4. Double check logs look good:

   oc exec -it $(oc get pod -o name | grep surveygeocoderservice) -c surveygeocoderservice -- cat /logs/messages.log
   

5. Create a KNative Eventing KafkaSource for surveyGeocoderService (if needed, replace bootstrapServers with the AMQ Streams Kafka Cluster bootstrap address):

   oc apply -f lib/example_surveygeocoderkafkasource.yaml
   

6. Query until OK is ++ for all lines:

   kn source kafka describe locationtopicsource
   

7. Note that scale-down-delay does not apply to the initial pod creation so the pod will be terminated about 30 seconds after it's initially created. Once a real user request is made to this application, then scale-down-delay will apply. Therefore, if you want to tail the logs of the pod, first wait for the initial pod to terminate and then make a request to the application and then you can tail the pod logs.

Update KNative Service to new image

Run kn service update with the service name and the same image name from the YAML; for examples:

kn service update surveyinputservice --image=image-registry.openshift-image-registry.svc:5000/libertysurvey/surveyinputservice

kn service update surveygeocoderservice --image=image-registry.openshift-image-registry.svc:5000/libertysurvey/surveygeocoderservice

kn service update surveyadminservice --image=image-registry.openshift-image-registry.svc:5000/libertysurvey/surveyadminservice

Test

1. Submit a location input:
   1. Using the command line:
      1. Execute:

         curl -k --data "textInput1=New York, NY" "$(kn service list surveyinputservice -o jsonpath="{.items[0].status.url}{'\n'}")/LocationSurvey"
         

      2. Check for a successful output:

         Your submission has been received
         

   2. Using the browser:
      1. Find and open the URL:

         kn service list surveyinputservice -o jsonpath="{.items[0].status.url}{'/location.html\n'}"
         

      2. Click Location Survey and submit the form
2. Double check logs look good:

   oc exec -it $(oc get pod -o name | grep surveygeocoderservice) -c surveygeocoderservice -- tail -f /logs/messages.log
   

Debugging

1. Tail surveyinputservice logs:

   oc exec -it $(oc get pod -o name | grep surveyinputservice) -c surveyinputservice -- tail -f /logs/messages.log
   

2. Tail surveyadminservice logs:

   oc exec -it $(oc get pod -o name | grep surveyadminservice) -c surveyadminservice -- tail -f /logs/messages.log
   

3. Tail surveygeocoderservice logs:

   oc exec -it $(oc get pod -o name | grep surveygeocoderservice) -c surveygeocoderservice -- tail -f /logs/messages.log
   

Clean-up tasks

lib/cleanup_all.sh

Delete surveyAdminService

1. Delete the KafkaSource:

   kn source kafka delete geocodetopicsource
   

2. Delete the KNative Service:

   kn service delete surveyadminservice
   

Delete surveyGeocoderService

1. Delete the KafkaSource:

   kn source kafka delete locationtopicsource
   

2. Delete the KNative Service:

   kn service delete surveygeocoderservice
   

Delete surveyInputService

kn service delete surveyinputservice

Testing Locally

Only some functions can be tested locally without KNative.

Testing surveyAdminService

1. Run surveyAdminService:

   podman run --privileged --rm -e GEOAPIFY_API_KEY=YOUR_KEY -p 8080:8080 -p 8443:8443 -it localhost/surveyadminservice:latest
   

2. Open browser to http://localhost:8080/geolocation.jsp
3. Post a CloudEvent:

   curl -X POST http://localhost:8080/api/cloudevents/geocodeComplete \
     -H "Ce-Source: https://example.com/" \
     -H "Ce-Id: $(uuidgen)" \
     -H "Ce-Specversion: 1.0" \
     -H "Ce-Type: CloudEvent1" \
     -H "Content-Type: text/plain" \
     -d "40.7127753 -74.0059728 New York, NY"
   

4. Switch back to the browser and you should see the point.

Testing surveyInputService

1. Create Kafka container network if it doesn't exist:

   podman network create kafka
   

2. Start Kafka if it's not started:

   podman run --rm -p 9092:9092 -e "ALLOW_PLAINTEXT_LISTENER=yes" -e "KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:9092" -e "KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093" -e "KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT" -e "KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-0:9093" -e "KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER" -e "KAFKA_CFG_PROCESS_ROLES=controller,broker" -e "KAFKA_CFG_NODE_ID=0" --name kafka-0 --network kafka docker.io/bitnami/kafka
   

3. Run surveyInputService:

   podman run --privileged --rm --network kafka  --rm -p 8080:8080 -p 8443:8443 -it localhost/surveyinputservice:latest
   

4. Wait for the message:

   [...] CWWKZ0001I: Application surveyInputService started [...]
   

5. Access http://localhost:8080/location.html or https://localhost:8443/location.html

Testing surveyGeocoderService

1. Create Kafka container network if it doesn't exist:

   podman network create kafka
   

2. Start Kafka if it's not started:

   podman run --rm -p 9092:9092 -e "ALLOW_PLAINTEXT_LISTENER=yes" -e "KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:9092" -e "KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093" -e "KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT" -e "KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-0:9093" -e "KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER" -e "KAFKA_CFG_PROCESS_ROLES=controller,broker" -e "KAFKA_CFG_NODE_ID=0" --name kafka-0 --network kafka docker.io/bitnami/kafka
   

3. Run surveyGeocoderService:

   podman run --privileged --rm -p 8080:8080 -p 8443:8443 -e "GEOAPIFY_API_KEY=INSERT_API_KEY" -it localhost/surveygeocoderservice:latest
   

4. Post a CloudEvent:

   curl -X POST http://localhost:8080/api/cloudevents/locationInput \
     -H "Ce-Source: https://example.com/" \
     -H "Ce-Id: $(uuidgen)" \
     -H "Ce-Specversion: 1.0" \
     -H "Ce-Type: CloudEvent1" \
     -H "Content-Type: text/plain" \
     -d "New York, NY"
   

Testing without containers

1. Change directory to the application
2. GEOAPIFY_API_KEY=INSERT_API_KEY mvn clean liberty:dev
3. Open http://localhost:8080/

Learn More

1. https://developer.ibm.com/articles/develop-reactive-microservices-with-microprofile/
2. https://openliberty.io/guides/microprofile-reactive-messaging.html
3. https://smallrye.io/smallrye-reactive-messaging/latest/concepts/concepts/
4. https://openliberty.io/blog/2022/10/17/microprofile-serverless-ibm-code-engine.html
5. OpenLiberty/open-liberty#19889
6. OpenLiberty/open-liberty#21659