Removing limits on Kafka controller

[matzew]

Fixes #457 (for Kafka). Thanks @nicolaferraro for filing the (generic) issue

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• contrib/kafka/OWNERS [matzew]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[duglin]

Why are we allowing pods to each up as much resources as they want?
It would be better to just raise the limit instead, if needed

[matzew]

I'd think that if we add those, we need to have confidence in the actual values - I don't see value in blind guesses, so to have some, just to have some ... See: https://bugzilla.redhat.com/show_bug.cgi?id=1714183

[duglin]

agreed to trying to pick smarter values, so let's grab the values used to fix that issue instead of just removing them

[evankanderson]

Limits (and requests) may also need to be dynamic based on cluster size/number of events in flight. Vertical Pod Autoscaling might be one way to set those limits dynamically.

[evankanderson]

/lgtm

[alanconway]

What is it we are trying to limit?

a) Ensure enough resources to get to some "standard" steady state of operation.
b) Ensure enough resources to maintain some desired scale/size/load.
c) Ensure no more resources than the system designer is willing to give it.

We can do a) in theory and it would be great if we provided scaling guides for people trying to figure out b) and c), but it's not trivial. We need to understand and measure relationship between scale, load and resources in our code and in our dependencies (Kafka libs etc.) I'm not saying we shouldn't, but it is extra documentation, testing and benchmarking, and it will impact the pace of innovation.