Google Pub/Sub Troubleshooting | OpenMetadata Connector

Troubleshoot common Google Pub/Sub connector issues in OpenMetadata — authentication failures, permission errors, schema registry problems, and ingestion issues.

Troubleshooting

Workflow Deployment Error

If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container.

• You can then Edit the Ingestion Pipeline and Deploy it again.
• From the Connection tab, you can also Edit the Service if needed.

Connector Debug Troubleshooting

This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios.

How to Enable Debug Logging for Any Ingestion

To enable debug logging for any ingestion workflow in OpenMetadata:

1. Navigate to Services Go to Settings > Services > Service Type (e.g., Database) in the OpenMetadata UI.

2. Select a Service Choose the specific service for which you want to enable debug logging.

3. Access Ingestion Tab Go to the Ingestion tab and click the three-dot menu on the right-hand side of the ingestion type, and select Edit.

4. Enable Debug Logging In the configuration dialog, enable the Debug Log option and click Next.

5. Schedule and Submit Configure the schedule if needed and click Submit to apply the changes.

Permission Issues

If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions.

Google Pub/Sub-Specific Issues

Authentication Fails: “Could not determine project ID”

Cause: The project ID cannot be resolved from the provided credentials or connection config.

Resolution: Explicitly set the projectId field in the connection configuration. This overrides the project ID read from the GCP credentials and is required when the service account belongs to a different project than the Pub/Sub topics.

Emulator Connection Refused

Cause: useEmulator is set to true but hostPort is still set to pubsub.googleapis.com (the default).

Resolution: When using the Pub/Sub emulator, set hostPort to the emulator address (e.g., localhost:8085). The connector will raise a validation error if the default production endpoint is used with useEmulator: true.

Schema Registry Topics Not Showing Schema

Cause: schemaRegistryEnabled is true but the Schema Registry client cannot be initialized when useEmulator is also true.

Resolution: Schema Registry is not supported in emulator mode. The schema client is automatically disabled when useEmulator: true. To use the Schema Registry, connect to the production Pub/Sub service.

Topics Missing from Ingestion

Cause 1: Topics are being filtered out by dead letter topic detection.

Resolution: Set includeDeadLetterTopics: true to include dead letter topics in metadata extraction.

Cause 2: Topic names are excluded by the topicFilterPattern.

Resolution: Review your topicFilterPattern configuration. To ingest all topics, set includes: [".*"] and leave excludes empty.

pubsub.subscriptions.list Permission Denied

Cause: The service account lacks the pubsub.subscriptions.list permission, which is required for dead letter topic detection and subscription metadata.

Resolution: Grant the roles/pubsub.viewer role to the service account, or add the pubsub.subscriptions.list permission individually. Alternatively, set includeSubscriptions: false and includeDeadLetterTopics: false to skip subscription-related API calls.