grpc single thread implementation with dialogflow cx finishing issue

environment: GRPC 1.28.1 Redhat 7 service used from google apis is dialogflow cx v3 ref: https://cloud.google.com/dialogflow/cx/docs/reference/rpc/google.cloud.dialogflow.cx.v3 language: CPP/C++ Compiler: GCC 8.3.1

What did you do?

I am trying to do async streaming of audio input from the microphone to dialogflow cx(server) using grpc single thread with completion queue and I follow this example, which is taken from speech api documentation https://github.com/GoogleCloudPlatform/cpp-samples the sample code is in -> cpp-samples/speech/api/streaming_transcribe_singlethread.cc

and my code flow is similar to that.

After creating all the resources including Channel, CompletionQueue, Context, streamer, Stub, etc but for Dialogflow CX service. if i have sent audio to the server but have not done with complete writing as the user was still speaking so far but now he hangs up while still speaking. how to unwind the grpc resources. I don't want to wait on any result from the server and want to abort the request quickly but smoothly.

At the moment i am following these steps to maintain graceful resource deallocation but it takes time to wait and server does not send grpc status NOT ok to send finish as shown in the example, so i am sending finish at step c) from the client side.

a) while audio bytes were written, stop further audio sending b) send Writes Done and wait for Writes_done complete c) send finish d) deallocate the resources

What actually are the steps to close the stream from client side Can i send finish and deallocate resources without reading completion queue for any messages in there? see below concerns

Some experiments i did which led to segmentation fault.

1. sometimes if the user was starting to speak and he hangs up without client code sending any audio bytes but sending Writes Done leads to segmentation fault 1) below

2. if finish is sent without waiting to read the completion queue for the Writes done, i see the segmentation fault 2) below

What did you expect to see?

smooth unwinding of grpc stack when i send Writes Done or Finish

What did you see instead?

segmentation fault

1) If i try to do WritesDone and try to finish i see this segmentation fault

    #0  0x00007f00e29763d7 in raise () from /lib64/libc.so.6      #1  0x00007f00e2977ac8 in abort () from /lib64/libc.so.6      #2  0x00007f007ceb3db4 in grpc::CoreCodegen::assert_fail (this=<optimized out>,      failed_assertion=<optimized out>, file=<optimized out>, line=<optimized out>)      at src/cpp/common/core_codegen.cc:237      #3  0x00007f007bdd0c1c in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpClientSendClose, grpc::internal::CallNoOp<4>,          grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::ContinueFillOpsAfterInterception (this=0x7f00b80520e8)      at /home/user/xs.7861e0.rh7/grpc-1.28.1.422777/include/grpcpp/impl/codegen/call_op_set.h:947      #4  0x00007f007bdd09a2 in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpClientSendClose, grpc::internal::CallNoOp<4>,          grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::FillOps (this=0x7f00b80520e8, call=0x7f00b8051e00)      at /home/user/xs.7861e0.rh7/grpc-1.28.1.422777/include/grpcpp/impl/codegen/call_op_set.h:868      #5  0x00007f007bdc82af in grpc::internal::Call::PerformOps (this=0x7f00b8051e00, ops=0x7f00b80520e8)      at /home/user/xs.7861e0.rh7/grpc-1.28.1.422777/include/grpcpp/impl/codegen/call.h:69      #6  0x00007f007bdd0fe8 in grpc_impl::ClientAsyncReaderWriter<google::cloud::dialogflow::cx::v3::StreamingDetectIntentRequest,          google::cloud::dialogflow::cx::v3::StreamingDetectIntentResponse>::WritesDone (      this=0x7f00b8051de0, tag=0x7f00b8212990)      at /home/user/xs.7861e0.rh7/grpc-1.28.1.422777/include/grpcpp/impl/codegen/async_stream_impl.h:584  

Is there any timer that runs for the allocated channel(resource) as i have seen sometimes if i don't follow the above steps to gracefully unwind, sometimes i see this coredump on my box.

    #0  0x00007fc7f45e3776 in publish_app_metadata (call=call@entry=0x7fc83c03b8a0,      is_trailing=is_trailing@entry=1, b=<optimized out>, b=<optimized out>)      at src/core/lib/surface/call.cc:1004      #1  0x00007fc7f45e61f3 in recv_trailing_filter (batch_error=<optimized out>, b=0x7fc83c03bce0,      args=0x7fc83c03b8a0) at src/core/lib/surface/call.cc:1084      #2  receiving_trailing_metadata_ready (bctlp=0x7fc83c0cc700, error=<optimized out>)      at src/core/lib/surface/call.cc:1528      #3  0x00007fc7f465e3cb in Run (location=..., error=0x0, closure=<optimized out>)      at ./src/core/lib/iomgr/closure.h:245      #4  grpc_core::SubchannelCall::RecvTrailingMetadataReady (arg=0x7fc83c071b80, error=0x0)      at src/core/ext/filters/client_channel/subchannel.cc:296      #5  0x00007fc7f46afa8f in Run (location=..., error=0x0, closure=<optimized out>)      at ./src/core/lib/iomgr/closure.h:245      #6  recv_trailing_metadata_ready (user_data=<optimized out>, error=0x0)      at src/core/ext/filters/message_size/message_size_filter.cc:235      #7  0x00007fc7f4612424 in Run (location=..., error=0x0, closure=<optimized out>)      at ./src/core/lib/iomgr/closure.h:245      #8  recv_trailing_metadata_ready (user_data=<optimized out>, error=0x0)      at src/core/ext/filters/http/client/http_client_filter.cc:212      #9  0x00007fc7f45c54e1 in exec_ctx_run (closure=<optimized out>, closure=<optimized out>, error=0x0)      at src/core/lib/iomgr/exec_ctx.cc:153      #10 grpc_core::ExecCtx::Flush (this=0x7fc7e88c5cf0) at src/core/lib/iomgr/exec_ctx.cc:153      #11 0x00007fc7f45c13b6 in pollset_work (pollset=0x7fc830208980, worker_hdl=<optimized out>,      deadline=<optimized out>) at ./src/core/lib/iomgr/exec_ctx.h:216      #12 0x00007fc7f464113c in run_poller (error=0x0, arg=0x7fc83016de30) at ./src/core/lib/iomgr/exec_ctx.h:216      #13 run_poller (arg=0x7fc83016de30, error=<optimized out>)      at src/core/ext/filters/client_channel/backup_poller.cc:114      #14 0x00007fc7f45c54e1 in exec_ctx_run (closure=<optimized out>, closure=<optimized out>, error=0x0)      at src/core/lib/iomgr/exec_ctx.cc:153      #15 grpc_core::ExecCtx::Flush (this=0x7fc7e88c5cf0) at src/core/lib/iomgr/exec_ctx.cc:153      #16 0x00007fc7f45d7b19 in run_some_timers () at ./src/core/lib/iomgr/exec_ctx.h:216      #17 timer_main_loop () at src/core/lib/iomgr/timer_manager.cc:237      #18 timer_thread (completed_thread_ptr=0x7fc8100218d0) at src/core/lib/iomgr/timer_manager.cc:284      #19 0x00007fc7f49ffaa3 in operator() (__closure=0x0, v=<optimized out>) at src/core/lib/gprpp/thd_posix.cc:140      #20 grpc_core::(anonymous namespace)::ThreadInternalsPosix::<lambda(void*)>::_FUN(void *) ()      at src/core/lib/gprpp/thd_posix.cc:145      #21 0x00007fc7f49ffaa3 in ThreadInternalsPosix (options=..., success=<optimized out>, arg=<optimized out>,      thd_body=<optimized out>, thd_name=<optimized out>, this=<optimized out>)      from /home/holly/lib/libs_grpc++.so      #22 grpc_core::Thread::Thread (this=<optimized out>, thd_name=<optimized out>, thd_body=<optimized out>,      arg=<optimized out>, success=<optimized out>, options=...) at src/core/lib/gprpp/thd_posix.cc:186  

https://stackoverflow.com/questions/66791347/grpc-single-thread-implementation-with-dialogflow-cx-finishing-issue March 25, 2021 at 08:28AM