Pubsub: Warning during acknowledgement message on subscription with enabled Exactly once delivery flag

[rozhkova-syberry]

Environment details

1. OS type and version: Windows 10
2. java version: 11
3. spring boot version: 2.6.7
4. google-cloud-pubsub library version: 1.115.1

Steps to reproduce

1. Create pubsub subscription on gcp with the following configurations:
   • Delivery type: Pull
   • Acknowledgment deadline: 600 secs
   • Exactly once delivery: True
   • Retry policy: retry immediately
2. Create spring boot application and add
   • 'com.google.cloud:spring-cloud-gcp-starter-pubsub' as a dependency . This dependency contains google-cloud-pubsub library
     1.115.1
   • mavenBom "com.google.cloud:spring-cloud-gcp-dependencies:3.1.0"
3. Configure PubSubConfiguration class to receive messages from specified subscription

Code

PubSubConfiguration class

    @Bean 
    public PubSubInboundChannelAdapter messageChannelAdapter(
        @Qualifier("inputChannel") MessageChannel inputChannel,
        PubSubTemplate pubSubTemplate
    ) {
        PubSubInboundChannelAdapter adapter =
            new PubSubInboundChannelAdapter(pubSubTemplate, subscriptionName);
        adapter.setOutputChannel(inputChannel);
        adapter.setAckMode(AckMode.MANUAL);
        adapter.setPayloadType(WorkflowEventMessage.class);
        adapter.setErrorChannelName(ERROR_CHANNEL_NAME);

        return adapter;
    }

    @Bean
    @Qualifier("inputChannel")
    public MessageChannel inputChannel() {
        return new DirectChannel();
    }

    @Bean
    @ServiceActivator(inputChannel = "inputChannel")
    public MessageHandler messageReceiver() {
        // do the logic
       originalMessage.ack();
    }

application.yaml

spring:
  cloud:
    gcp:
      pubsub:
        subscriber:
          max-ack-extension-period: 600

Actual behavior
Periodically I see in the logs warning

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed

After investigation I found that this exception appears after modifyAckDeadline call during acknowledgement pubsub message.

Expected behavior
No warnings in the logs.

Additional information
Pubsub messages are processed successfully without redeliveries but with this warning in the logs.
I updated google-cloud-pubsub library to 1.120.12 and tested it, but I still saw this warning in the logs.
If Exactly once delivery is disabled on subscription than there are no warnings in the logs.
Why this warning is displayed and and what issues it can potentially cause?

[samcrutt9900]

I'm experiencing exactly the same issue as this using 1.123.12 of the client lib and subscribing to an exactly once delivery subscription.

[maitrimangal]

Might be fixed with #1540

[maitrimangal]

Just merged this PR #1540 , please reopen this issue if the problem you are having still reoccurs.

[Sourc]

I've seen this issue as well when looking into another issue #1778, so it still seems to be there. It happens separately from the concurrent modification issues.

[kamalaboulhosn]

@Sourc What is the version of Spring and the client library you are using?

[Sourc]

@kamalaboulhosn No Spring, same setup as described in #1778. But I actually believe my warnings are caused by the soft-crashing of the subscribers, so it's most likely different from what's been the cause here. No need to re-open this issue just yet.