KafkaConnector error: 500 Internal Server Error

[urcaly00]

Hello,
We recently started looking at strimzi and are running into errors with the KafkaConnector.
Any suggestions to help troubleshoot or resolve them are welcome. Below is one example.

Thank you,
Lavaughn

strimzi 0.24
kafka 2.8.0
io.debezium.connector.mysql.MySqlConnector
Shared cluster and space does not appear to be an issue

ERROR
'GET /connectors/xyz-connector/config returned 500 (Internal Server Error): Request timed out'
reason: ConnectRestException

KafkaConnect

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.8.0
  image: image1
  replicas: 1
  bootstrapServers: lab:9092
  config:
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    group.id: group1
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
  externalConfiguration:
    env:
      - name: DKR_PROXY
        valueFrom:
          secretKeyRef:
            name: dkrproxy
            key: dkr_proxy
      - name: HTTP_PROXY
        valueFrom:
          secretKeyRef:
            name: httpproxy
            key: http_proxy
      - name: HTTPS_PROXY
        valueFrom:
          secretKeyRef:
            name: httpsproxy
            key: https_proxy
      - name: NO_PROXY
        valueFrom:
          secretKeyRef:
            name: noproxy
            key: no_proxy
  template:
    pod:
      imagePullSecrets:
        - name: lab-registry
  logging:
    type: inline
    loggers:
      connect.root.logger.level: "INFO"
      log4j.rootLogger: "INFO"

KafkaConnectorYAML

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: xyz-connector
  labels:
    strimzi.io/cluster: cluster1
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1  
  config:
    database.history.kafka.bootstrap.servers: lab:9092
    database.history.kafka.topic: sql-topic
    database.hostname: 1.1.1.1
    database.include.list: dbname
    database.user: xyzname
    database.password: xyz
    database.port: 3000
    database.server.id: 1333
    database.server.name: dbznm
    include.schema.changes: true
    table.include.list: table1
    topic.creation.enable: true

One interesting and unrelated note: With multiple connect clusters there was extensive log info available from the service connect api. With only one connect cluster no log files were generated until I bumped the logging level to DEBUG in the KafkaConnect

[scholzj]

As mentioned before - you should double check if there is no other Connect cluster using the same internal topics and group: https://strimzi.io/docs/operators/0.24.0/full/deploying.html#con-kafka-connect-multiple-instances-str.

I would also make sure to set resources for your Connect cluster. When I use Connect, I normally give it 0.5-1 CPU and 1-2 Gi of memory. But I'm not sure how much Debezium really needs. It might need more.

Maybe you can also try to use some simpler connectors to see what the problem relates to - e.g. https://github.com/strimzi/strimzi-kafka-operator/blob/0.24.0/examples/connect/source-connector.yaml or I use this for testing: https://github.com/scholzj/echo-sink

[urcaly00]

Hi Jakub, Thanks for your response. I have only one connect cluster now but the problem persists. I will try to increase the resources specifically for the connector. We have tried the Sink and Source Filestream connectors and they work fine.
Thank you,
Lavaughn

[scholzj]

The connector does not have its resources. They are always on the Connect level. So you have to set them in the KafkaConnect custom resource. If the small connectors work and Debezium doesn't, it should be worth trying since Debezium is definitely heavier connector (it also does a lot of stuff compared to them so it is expected). If it doesn't help, you should probably share the full log form Connect.

[urcaly00]

Thanks Jakub, I have some new errors. How can I use the validate function of the connect-api?

So far:
I added resources at the Connect level.

  resources:
    requests:
      cpu: "1"
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 1Gi

Current error:
Log Error
ERROR Failed testing connection for jdbc:mysql://servername:port?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL&connectTimeout=30000 with user 'username' (io.debezium.connector.mysql.MySqlConnector) [DistributedHerder-connect-1]

Connector error
message: |-
PUT /connectors/dbz-connector/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 1 error(s):
Unable to connect: Communications link failure

  The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
reason: ConnectRestException
status: "True"
type: NotReady

From the kubernetes shell (Connect API)

slookup sqlservername:port is successful
curl connect-api:8083/connectors
returns an empty [] (can't find the dbz-connector
Also, username and password is correct.

FYI: If I try a JDBC connector, it does show up if I run curl connect-api:8083/connectors from the kubernetes shell

[scholzj]

Well, as the error suggests, you can use the validate endpoint to validate the configuration. But I'm not sure how much information does that actually give you. If the error is Unable to connect I'm not sure how much can the validation help.

[urcaly00]

I'm past the connectivity issue now. From the logs I can see a list of tables in the database so it's reading the database. I may need different configuration so a config validation would be helpful.

Current Connector Error:
message: 'GET /connectors/dbz-connector/topics returned 404 (Not Found):
Unexpected status code'
reason: ConnectRestException
status: "True"

From the kubernetes shell, I tried
curl connect-api:8083/connectors/dbz-connector/config/validate
I get a 404, is this the right location for the validate function?

I also tried

curl -s -X PUT -H "Content-Type:application/json"
http://connector-api:8083/connectors/dbz-connector/config/validate/
-d '{"connector.class":"io.debezium.connector.sqlserver.SqlServerConnector","database.dbname":"dbname","database.user":"dbuser","database.server.id":"1333","tasks.max":"1","database.history.kafka.bootstrap.servers":"kafka:9092","database.history.kafka.topic":"hist-topiic","database.server.name":"dbznm","database.port":"3333","include.schema.changes":"true","topic.prefix":"dbx","database.hostname":"sqlservername","database.password":"sqlserverpassword","name":"dbz-connector","table.include.list":"dbo.tablename"}'
No response

[scholzj]

I don't know, I do not use the validate functiona normally. The Kafka docs should help you, they list all the endpoints.

[urcaly00]

So far, I'm not able to find anything pertinent about validate, if you have any links I would appreciate it.
For the cluster, I was able to get some more memory but not CPU.
The topics error still remains
message: 'GET /connectors/dbz-connector/topics returned 404 (Not Found):
Unexpected status code'
reason: ConnectRestException
status: "True"

Progress:
The connector is set to watch a table, and with mode set to incrementing the source connector created a topic in the cluster (prefix + a random identifier) after a record was added to the table.

I can't tell if there is any data in the topic.
I tried running the consumer.sh but nothing shows up.

Is there configuration to specify a target topic name and avoid the odd looking topic name?
So far I haven't found it.

[scholzj]

Is there configuration to specify a target topic name and avoid the odd looking topic name?
So far I haven't found it.

I think that is up to the connector?

The topics error still remains

I guess that suggests the connector is not properly created?

[urcaly00]

I was able to get it working but the error message remains.

message: 'GET /connectors/dbz-connector/topics returned 404 (Not Found):
Unexpected status code'
reason: ConnectRestException
status: "True"
type: NotReady

[scholzj]

Ok, so, what Strimzi version are you using and how did you build your Kafka Connect image with the plugins?

[urcaly00]

Strimzi 0.24
Image built using Docker and I used kafka-connect-jdbc-10.3.3.jar

[scholzj]

But how exactly did you built the image? Can you share the Dockerfile?

[urcaly00]

Thanks! I'm guessing I need something to match my version.
I used the Dockerfile info from https://strimzi.io/blog/2020/05/07/camel-kafka-connectors/ and it references a different version of strimzi/kafka than we have. For our versions strimzi/kafka:0.24.0-kafka-2.8.0 but there was no manifest found.

--
FROM strimzi/kafka:0.17.0-kafka-2.4.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/camel
COPY ./kafka-connect-jdbc-10.3.3.jar /opt/kafka/plugins/camel/

[scholzj]

But you cannot use the base image from Strimzi 0.17 if you use Strimzi 0.24. You have to use the matching base image from Strimzi 0.24.