Duplex Issues?

Mar 2, 2012 at 10:03 AM

Hi Kerry,

I've implemented a acknowledgement pattern in my server which requires the client to send back an acknowledgement for every message sent by the server. This way we can guarantee delivery. The send process waits on the acknowledgement for a set amount of time (notionally 5 seconds) and if the ack is not received in that time - the session is deemed to be disconnected and is closed. Simple.

However, I'm seeing some strange behaviour when implementing the acknowledgement response in the client. 

What is happening is that the server sends a data payload to the client using the session's SendResponseAsync() method, the client picks it up in it's DataReceived event and sends back using the Send(byte[],....) method an acknowledgement. The server doesn't seem to receive the acknowledgement from the client. I originally though it was because I was sending data in the DataReceived event on the client, but I changed it to let the event complete and then send from another thread but the problem still exists.

New connections are ok and can received data - it's only when they receive data from the server and return an acknowledgement that the server fails to receive data.

Am I not understanding something here?

Thanks

Coordinator
Mar 2, 2012 at 10:32 AM
How do you send the data? Are you using websocket4net?

Sent from my Windows Phone

From: ChubbyArse
Sent: 3/2/2012 6:03 PM
To: kerry-jiang@hotmail.com
Subject: Duplex Issues? [SuperWebSocket:346994]

From: ChubbyArse

Hi Kerry,

I've implemented a acknowledgement pattern in my server which requires the client to send back an acknowledgement for every message sent by the server. This way we can guarantee delivery. The send process waits on the acknowledgement for a set amount of time (notionally 5 seconds) and if the ack is not received in that time - the session is deemed to be disconnected and is closed. Simple.

However, I'm seeing some strange behaviour when implementing the acknowledgement response in the client.

What is happening is that the server sends a data payload to the client using the session's SendResponseAsync() method, the client picks it up in it's DataReceived event and sends back using the Send(byte[],....) method an acknowledgement. The server doesn't seem to receive the acknowledgement from the client. I originally though it was because I was sending data in the DataReceived event on the client, but I changed it to let the event complete and then send from another thread but the problem still exists.

New connections are ok and can received data - it's only when they receive data from the server and return an acknowledgement that the server fails to receive data.

Am I not understanding something here?

Thanks

Mar 2, 2012 at 10:35 AM
Edited Mar 2, 2012 at 10:36 AM

Yes - using WebSocket4Net.

I'm calling

_socket.Send(data, 0, data.Length);

in the DataRecieved event (after I've processing the data received.

Coordinator
Mar 2, 2012 at 10:45 AM
How do you know the server didn;t receive the data?

From: [email removed]
Sent: Friday, March 02, 2012 6:35 PM
To: [email removed]
Subject: Re: Duplex Issues? [SuperWebSocket:346994]

From: ChubbyArse

Yes - using WebSocket4Net.

I'm calling

_socket.Send(data, 0, data.Length);

in the OnDataRecieved event (after I've processing the data received.

Mar 2, 2012 at 10:50 AM

I am seeing the same behaviour when using the MS client too.

The server is setup with Mode = SocketMode.Async

Thanks

Coordinator
Mar 2, 2012 at 10:51 AM
Please answer my last question.

From: [email removed]
Sent: Friday, March 02, 2012 6:50 PM
To: [email removed]
Subject: Re: Duplex Issues? [SuperWebSocket:346994]

From: ChubbyArse

I am seeing the same behaviour when using the MS client too.

The server is setup with Mode = SocketMode.Async

Thanks

Mar 2, 2012 at 10:54 AM

The server logging doesn't log anything after the original send from the server.

I've also run the server in debug and put a breakpoint on the NewDataReceived event. I can then see the initial "hello" come in from the client  - the server responds to this, but then the client's acknowledgement of the servers message doesn't arrive.

Thanks Kerry

Coordinator
Mar 2, 2012 at 10:57 AM

Did you handler the server's event "NewDataReceived"?

Coordinator
Mar 2, 2012 at 10:59 AM

The test case "SendDataTest" in the class below is to verify binary data transferring:

http://superwebsocket.codeplex.com/SourceControl/changeset/view/73515#1384207

Mar 2, 2012 at 11:12 AM

I handle both the NewDataReceived and NewMessageReceived on the server side. To map out the process that is causing this issue.

  1. The client initiates the connection and sends it identifying data over the connection.
  2. The server recieves this initial data on the NewDataReceived event and responds to the client with an payload that contains a correlationId.
  3. The client picks this up in it's DataReceived event and immediately responds using the Send method with an Acknowledgement message that contains the correlationId.
  4. The server *should* then pick up this data in the NewDataReceived event (the same event wired up that recieved the initial clients data in step 1) and process it (this allows us to guarantee the server message was received - cause it contains the same correlationId).

The problem I have though is that although the server gets the initial request (step1), after sending a message to it it fails to receive the data in Step 4.

All the time this is happening - new clients are able to connect and initiate the same process above - so it's not like the servers receive is locked - it just looks like on a per-session basis you can't receive at the server side after sending to the client (asynchronously).

Thanks

Coordinator
Mar 2, 2012 at 11:18 AM

Actually, receiving and sending are two different procedures and they won't interfere each other in SuperWebSocket.

Coordinator
Mar 2, 2012 at 11:21 AM

Could you write a test case like the url:

http://superwebsocket.codeplex.com/SourceControl/changeset/view/73515#1384207

So that I can produce it in my local then check the reason.

Mar 2, 2012 at 11:35 AM

OK - I'm a bit closer to understanding what is going on here. I think the session is being locked on the server side whilst I send out the message.

I'm using a derived server and session using the abstract classes WebSocketServer<> and WebSocketSession<>. Nice touch by the way! :-)

See below:

 

 public bool SendResponseSync(string correlationId, string data)
        {
            using (MethodLogger.Log("DeviceCommsChannelSession.SendResponseSync"))
            {               
                ManualResetEvent acknowledgementSignal = new ManualResetEvent(false);
                
                DataSent.TryAdd(correlationId, acknowledgementSignal);

                //Send the data down the channel
                SendResponseAsync(data);

                //Wait for a timely acknowledgement
                Boolean acknowledged = acknowledgementSignal.WaitOne(5000);

                //Remove the signal - it's been acknowledged
                DataSent.TryRemove(correlationId, out acknowledgementSignal);

                //Close the session if the acknowledgement was not recieved
                if (!acknowledged)
                {
                    LoggingManager.LogWarning("Did not get a timely response from the recipient - closing session down");
                    IObjectBuilder objectBuilder = new ObjectBuilder();
                    IDeviceCommsSessionManager sessionManager = objectBuilder.BuildObject<IDeviceCommsSessionManager>();
                    sessionManager.CloseSession(this);
                }

                return acknowledged;
                
            }
        }

Each session has a ConcurentDictionary that holds a ManualResetEvent and is keyed with the correlationId for each send the server does.

When data is sent the following happens:

  1. An ManualResetEvent is created and added to the dictionary keyed on the correlationId of the data being sent out.
  2. The message is sent
  3. The method waits on the ManualResetEvent for 5 seconds....
  4. Meanwhile the client gets the data and replies with the acknowledgement....
  5. The server *should* pick up the data on another thread and process it, where it looks in the dictionary and for the ManualResetEvent and calls the .Set().
  6. This releases the method waiting on the original thread (in step 3) and we then know that the message was sent and received.
  7. If the wait in Step 3 times out after 5 seconds then the message was not received and we close down the session.

What I think is happening is that the ManualResetEvent that halts the progress of the sessions SendResponseSync method is "locking" the session, and when the data comes in at the Socket level - SuperSockets is unable to get the lock on the session and bubble up the NewDataReceived?

Does that sound plausible. If so - are we able to make the sessions more thread safe?

Coordinator
Mar 2, 2012 at 11:41 AM
OHHH!
The code looks bad to get good performance.
Please always write your logic in async mode instead of use signal to sync....

From: [email removed]
Sent: Friday, March 02, 2012 7:35 PM
To: [email removed]
Subject: Re: Duplex Issues? [SuperWebSocket:346994]

From: ChubbyArse

OK - I'm a bit closer to understanding what is going on here. I think the session is being locked on the server side whilst I send out the message.

I'm using a derived server and session using the abstract classes WebSocketServer<> and WebSocketSession<>. Nice touch by the way! :-)

See below:

 public bool SendResponseSync(string correlationId, string data)
        {
            using (MethodLogger.Log("DeviceCommsChannelSession.SendResponseSync"))
            {               
                ManualResetEvent acknowledgementSignal = new ManualResetEvent(false);
                
                DataSent.TryAdd(correlationId, acknowledgementSignal);

                //Send the data down the channel
                SendResponseAsync(data);

                //Wait for a timely acknowledgement
                Boolean acknowledged = acknowledgementSignal.WaitOne(5000);

                //Remove the signal - it's been acknowledged
                DataSent.TryRemove(correlationId, out acknowledgementSignal);

                //Close the session if the acknowledgement was not recieved
                if (!acknowledged)
                {
                    LoggingManager.LogWarning("Did not get a timely response from the recipient - closing session down");
                    IObjectBuilder objectBuilder = new ObjectBuilder();
                    IDeviceCommsSessionManager sessionManager = objectBuilder.BuildObject<IDeviceCommsSessionManager>();
                    sessionManager.CloseSession(this);
                }

                return acknowledged;
                
            }
        }

Each session has a ConcurentDictionary that holds a ManualResetEvent and is keyed with the correlationId for each send the server does.

When data is sent the following happens:

  1. An ManualResetEvent is created and added to the dictionary keyed on the correlationId of the data being sent out.
  2. The message is sent
  3. The method waits on the ManualResetEvent for 5 seconds....
  4. Meanwhile the client gets the data and replies with the acknowledgement....
  5. The server *should* pick up the data on another thread and process it, where it looks in the dictionary and for the ManualResetEvent and calls the .Set().
  6. This releases the method waiting on the original thread (in step 3) and we then know that the message was sent and received.
  7. If the wait in Step 3 times out after 5 seconds then the message was not received and we close down the session.

What I think is happening is that the ManualResetEvent that halts the progress of the sessions SendResponseSync method is "locking" the session, and when the data comes in at the Socket level - SuperSockets is unable to get the lock on the session and bubble up the NewDataReceived?

Does that sound plausible. If so - are we able to make the sessions more thread safe?

Coordinator
Mar 2, 2012 at 11:45 AM

Please tell me your raw requirement? I do not quite understand your logic.

Mar 2, 2012 at 12:10 PM

OK- the raw requirement is to ensure that messages are received by the client.

As we have control over both the server and the client we can stipulate that any server message received by the client will have a correlationId which the client uses to send back and acknowledge that the message was indeed received. If the message is not received then the server application will take other steps (this is part of a bigger application).

As I have derived my session - it is sensible to have a method in that session that sends synchronously - by suspending the new SendResponseSync method (I pasted earlier) until the server has received the acknowledgement (on another thread) and releases the ManualResetEvent associated with the orignial send.

If we don't do this, in the event of a session being closed by the client, the SendResponse method will complete and we will be unaware that the message didn't get through until later when we get a SessionClosed event - and even then we don't know which messages got through and which ones didn't.

By the client acknowledging each message - we can guarantee that messages get through and mitigate when they don't.

Is there anyway that the session can be suspended without locking it down? I'm building a unit test to demonstrate the issue.

Coordinator
Mar 2, 2012 at 12:30 PM

Sending method in session:

public bool SendResponseSync(string correlationId, string data)
        {
                this.CorrelationId = correlationId;
                //Send the data down the channel
                SendResponseAsync(data);
        }

NewDataReceived handling:
//check the new received correlationID and the session.CorrelationID
// if the id is matched, go to next step

Enable clearIdleSession function, then if there is no request from clinet for a long time, the connection will be disconnected by server autmatically
Mar 2, 2012 at 12:44 PM

Thanks Kerry

We don't want to clear up idle connections. In fact during testing we've seen a connection live for more than 11 hours with no activity. It's a requirement of our server to maintain idle connections for an indefinite amount of time.

In your example SendResponseSync will always complete and not indicate if the call failed? We have to wait for the response from the client to allow the method to complete?

Coordinator
Mar 2, 2012 at 12:50 PM

It seems that you close the connection if no callback received in a time.

Yes, that method always complete. Your code needn't WAIT, just react when the server receive the client's callback.

Mar 2, 2012 at 12:58 PM
Edited Mar 2, 2012 at 12:58 PM

Our application however requires the process to be synchronous. If the message is not received, then the sending process needs to know - we can't defer until later when the acknowledgement comes in....

I'll hopefully be able to get a unit test to you to demonstrate that the session is not thread safe - it is locked by the send.

Coordinator
Mar 2, 2012 at 1:07 PM

Our application however requires the process to be synchronous.

May be wrong, that's your code's requirement. You must improved it.

Actually, you cannot know whether the message is received, you only can know whether the server receive the client's callback in one time span.

Please implement the functionality follow my instruction.

You can wait the callback in a new thread to finish the function in your way, I recommend you don't do it.