Introduction

Conduktor Gateway is a vendor-agnostic Apache Kafka proxy that has two options to run as a docker image, one which is distroless and one which has a distro, see our docker hub here.

As Gateway is deployed as part of a distributed system, a proxy for your Apache Kafka, we have some suggested best practices when opening a support case. Please follow these best practices to help us be as efficient as possible in resolving your support case with Gateway.

Best practices

State the background of the issue you are having so that we can go about reproducing the issue along with the version of Gateway you are using and what you expect to be happening to consider it to be working.

We also ask for the following pieces of information to help us troubleshoot:

• Ideally, capture the logs from the startup of Conduktor Gateway or restart one instance, this provides us with the version and specific settings you are using.
• The logs from the Gateway instances from approximately when the issue was observed.
  • If you are using more than one Gateway instance we will need logs from them all.
• If the issue is with a client application along with Conduktor Gateway we will need the logs from this client at approximately the time of issue, along with the version of the client library used.
  • For example, using Python 2.7 with confluent-kafka-python.
• Are you using Topic mappings, yes/no?
• Have you configured your Kafka clients to use transactions, yes/no?
• Are you using particular interceptors?

How to get logs from Gateway?

This depends on how you have started the docker image of Conduktor Gateway. Please find links to documentation and examples for Docker Compose and Kubernetes deployment methods. If you are using a managed container service from a Cloud provider it will depend on how you have set up your logging.

If you are using Docker Compose you can use the command as shown in Docker documentation here:

docker compose logs [OPTIONS] [SERVICE...]

For example, docker compose logs conduktor-gateway

If you are using Kubernetes, see the documentation here:

kubectl logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]

For example,  kubectl logs conduktor-gateway

Example of Startup Logs

Version 3.0.0 1444ec0437c6e8687a7be677542c0bff99e2aa4a 2024-03-20T16:35:59+0000

Computed configuration :
---
gatewayClusterId: "gateway"
gatewayRackId: null
gatewayGroupId: null
kafkaSelector: !<env>
prefix: "KAFKA_"
hostPortConfiguration:
bindHost: "0.0.0.0"
advertisedHost: "conduktor-gateway"
hostPrefix: "broker"
portCount: null
portStart: 6969
tenantInHostname: false
routing: "port"
authenticationConfig:
securityProtocol: "SASL_PLAINTEXT"
sslConfig:
keyStore:
keyStorePath: null
keyStorePassword: null
keyPassword: null
keyStoreType: "jks"
incomplete: true
trustStore:
trustStorePath: null
trustStorePassword: null
trustStoreType: "jks"
clientAuth: "NONE"
sslPrincipalMappingRules: null
incomplete: true
updateContextIntervalMinutes: 5
connectionsMaxReauthMs: 0
oauth:
jwksUrl: "https://login.microsoftonline.com/common/discovery/keys"
expectedIssuer: "https://sts.windows.net/38755287-df00/"
expectedAudiences:
- "00000002-0000-0000-c000-000000000000"
refresh: null
retry: null
maxRetry: null
scopeClaimName: null
subClaimName: null
authorizationCacheMs: 5000
mandatoryVCluster: false
httpConfig:
port: 8888
securedMetrics: true
superUsers:
- username: "admin"
password: "***"
admin: true
- username: "app-admin"
password: "***"
admin: true
threadConfig:
downStreamThread: 2
upstream:
numberOfThread: 0
maxPendingTask: 2147483647
maxIdleProducerMs: 600000
upstreamConnectionConfig:
numOfConnection: 10
maxIdleTimeMs: 200000
connectionPoolType: "NONE"
userPoolConfig:
jwt:
secretKey: "***"
signatureAlgorithm: "HS256"
defaultReAuthIntervalMs: 604800000
aclStoreConfig:
allowEveryoneIfNoAclFound: false
interceptorConfig:
configLocation: null
restrictingGlobalInterceptors: null
licenseConfig:
keyAlgorithm: "EC"
publicKey: "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEMOpFpNv+/=="
licenseKey: "***"
featureFlags:
restrictingTenantApis: false
audit: true
analytics: true
internalLoadBalancing: true
inFlightRequestExpiryMs: 30000
auditLogConfig:
specVersion: "0.1.0"
topic: null
numPartitionsOfTopic: -1
replicationFactorOfTopic: -1
analytics:
endpoint: "https://events.eu1.segmentapis.com"
writeKey: "sFyxHHaoDYlnhiFlFmMYu8YWnFczG95Z"
profiling: false
gaugeBackendBrokersTimerConfig:
initialDelayMs: 1000
delayMs: 60000
storage:
type: "KAFKA"
kafka:
defaultKCacheReplicationFactor: -1
expiryTtlMs: 604800000
topics:
topicMappings: null
interceptorConfigs: null
acls: null
consumerOffsets: null
consumerSubscriptions: null
license: null
userMappings: null
encryptionConfigs: null
securityProvider: "DEFAULT"

Example of Gateway Instance Logs

conduktor-gateway | 2024-04-08T09:19:30.019+0000 [ main] [INFO ] [ProxyConfiguration:176] - Using default JRE Security Provider
conduktor-gateway | 2024-04-08T09:19:30.159+0000 [ main] [INFO ] [EnvKafkaPropertiesProvider:17] - Loading Kafka Config from environment vars with prefix: KAFKA_
conduktor-gateway | 2024-04-08T09:19:30.199+0000 [ main] [INFO ] [EnvKafkaPropertiesProvider:31] - Adding Kafka config: security.protocol=SASL_PLAINTEXT via security.protocol env variable
conduktor-gateway | 2024-04-08T09:19:30.201+0000 [ main] [INFO ] [EnvKafkaPropertiesProvider:29] - Adding Kafka config: sasl.jaas.config=[hidden] via sasl.jaas.config env variable
conduktor-gateway | 2024-04-08T09:19:30.202+0000 [ main] [INFO ] [EnvKafkaPropertiesProvider:31] - Adding Kafka config: sasl.mechanism=PLAIN via sasl.mechanism env variable
conduktor-gateway | 2024-04-08T09:19:30.202+0000 [ main] [INFO ] [EnvKafkaPropertiesProvider:31] - Adding Kafka config: bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092 via bootstrap.servers env variable
conduktor-gateway | 2024-04-08T09:19:30.247+0000 [ main] [INFO ] [ClusterConfigurationService:37] - cluster configuration:
conduktor-gateway | ---
conduktor-gateway | security.protocol: "SASL_PLAINTEXT"
conduktor-gateway | sasl.mechanism: "PLAIN"
conduktor-gateway | sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"************\";"
conduktor-gateway | bootstrap.servers: "kafka-1:9092,kafka-2:9092,kafka-3:9092"
conduktor-gateway |
conduktor-gateway | 2024-04-08T09:19:35.618+0000 [ main] [INFO ] [BrokerManagerWithPortMapping:59] - GATEWAY_PORT_COUNT is not set, portCount has been defined as 5. We recommend to specify this value for production use.
conduktor-gateway | 2024-04-08T09:19:42.347+0000 [Vavr-executor-3] [WARN ] [NetworkClient:1145] - [Consumer clientId=kafka-cache-reader-_conduktor_gateway_consumer_subscriptions, groupId=kafka-cache-conduktor-gateway] Error while fetching metadata with correlation id 4 : {_conduktor_gateway_consumer_subscriptions=LEADER_NOT_AVAILABLE}
conduktor-gateway | 2024-04-08T09:19:44.045+0000 [ main] [INFO ] [PersistentLicenseService:94] - Applied new license with expiry: 2025-01-01T00:00:00.000Z
conduktor-gateway | 2024-04-08T09:19:56.910+0000 [conduktor_gateway-0] [INFO ] [ConsumerGroupMembershipService:138] - Update role to LEADER on _conduktor_gateway_license (conduktor_gateway)
conduktor-gateway | 2024-04-08T09:19:56.909+0000 [ main] [WARN ] [ConsumerGroupMembershipService:83] - Membership detected but no partition assigned (seems that we are restarting as an existing follower). Inspecting membership to define role
conduktor-gateway | 2024-04-08T09:19:57.033+0000 [ main] [INFO ] [ConsumerGroupMembershipService:92] - Registered in consumer group conduktor_gateway on _conduktor_gateway_license as NodeEndpoint[host=conduktor-gateway, port=6969, rack=null] with role LEADER in 2s
conduktor-gateway | 2024-04-08T09:19:57.848+0000 [ main] [INFO ] [ProxyBrokers:79] - Proxy channel bound: /0.0.0.0:6969
conduktor-gateway | 2024-04-08T09:19:57.867+0000 [ main] [INFO ] [ProxyBrokers:79] - Proxy channel bound: /0.0.0.0:6970
conduktor-gateway | 2024-04-08T09:19:57.869+0000 [ main] [INFO ] [ProxyBrokers:79] - Proxy channel bound: /0.0.0.0:6971
conduktor-gateway | 2024-04-08T09:19:57.872+0000 [ main] [INFO ] [ProxyBrokers:79] - Proxy channel bound: /0.0.0.0:6972
conduktor-gateway | 2024-04-08T09:19:57.873+0000 [ main] [INFO ] [ProxyBrokers:79] - Proxy channel bound: /0.0.0.0:6973
conduktor-gateway | 2024-04-08T09:19:57.910+0000 [ main] [INFO ] [BrokerManagerWithPortMapping:127] - Initializing proxy 6969 for node KafkaNode(clusterId=main, node=kafka-1:9092 (id: 1 rack: null))
conduktor-gateway | 2024-04-08T09:19:57.914+0000 [ main] [INFO ] [BrokerManagerWithPortMapping:121] - Gateway started successfully with port range 6969-6973
conduktor-gateway | 2024-04-08T09:20:01.126+0000 [vert.x-eventloop-thread-0] [INFO ] [AppVerticle:74] - Proxy HTTP server started on port 8888
conduktor-gateway | 2024-04-08T09:20:28.093+0000 [vert.x-eventloop-thread-0] [INFO ] [PersistentAuditLog:101] - API POST /admin/userMappings/v1
conduktor-gateway | 2024-04-08T09:20:28.181+0000 [vert.x-eventloop-thread-0] [INFO ] [PersistentAuditLog:101] - API POST /admin/userMappings/v1
conduktor-gateway | 2024-04-08T09:20:28.224+0000 [vert.x-eventloop-thread-0] [INFO ] [PersistentAuditLog:101] - API GET /admin/userMappings/v1