How to send large amount of data

Apr 1, 2012 at 7:12 AM

Hi,

I modified the NUnit test program to send large amount of data. But, I can only send very small amount of data, like few thousands of byte (not 100% sure what's the limit).

All NUnit test seems sending small amount of data. Not more than few thousands, I guess.

This is what I have done (the following is partial code based on NUnit test):

1) Change client code to generate large number of GUID string, and comment out 3 lines of code so that client will send more data to server.

2) I found that if changing the loop of generating GUID string to 200, the server handler m_WebSocketServer_NewDataReceived is not able to get response. The data client sending is just few thousands of byte. If decrease the number, such as 100, server handler works.

3) I am using SuperSocket v1.4.4. SuperWebSocket v0.4.0.

Since I just played for few days of SuperWebSocket, I am not sure if my  understanding of how SuperWebSocket sending large amont of data works is correct. Please help:

a) Am I using wrong version? As I saw a post about 4 months ago with similar issue, and you said the issue is already fixed.

b) If the method I am using is correct? Can you point me to a good sample of sending large amount of data, such as 10 MB. Same API works?

 

 

 

// This is server

        [TestFixtureSetUp]
        public virtual void Setup()
        {
            LogUtil.Setup(new ConsoleLogger());

            m_WebSocketServer = new WebSocketServer(new BasicSubProtocol("Basic", new List<Assembly> { this.GetType().Assembly }));
            m_WebSocketServer.NewDataReceived += new SessionEventHandler<WebSocketSession, byte[]>(m_WebSocketServer_NewDataReceived);
            m_WebSocketServer.Setup(new RootConfig(), new ServerConfig
            {
                Port = 81,
                Ip = "Any",
                MaxConnectionNumber = 100,
                Mode = SocketMode.Async,
                Name = "SuperWebSocket Server"
            }, SocketServerFactory.Instance);
        }

        void m_WebSocketServer_NewDataReceived(WebSocketSession session, byte[] e)
        {
            //Echo
            session.SendResponse(e);
        }

.........

// This client

        [Test, Repeat(10)]
        public void SendDataTest()
        {
            WebSocket webSocketClient = new WebSocket(string.Format("ws://127.0.0.1:{0}/websocket", m_WebSocketServer.Config.Port), "basic", m_Version);
 
            if (!webSocketClient.SupportBinary)
                return;

            webSocketClient.Opened += new EventHandler(webSocketClient_Opened);
            webSocketClient.Closed += new EventHandler(webSocketClient_Closed);
            webSocketClient.DataReceived += new EventHandler<DataReceivedEventArgs>(webSocketClient_DataReceived);
            webSocketClient.Open();

            if (!m_OpenedEvent.WaitOne(1000))
                Assert.Fail("Failed to Opened session ontime");

            StringBuilder sb = new StringBuilder();

  //          for (int i = 0; i < 10; i++)    // This is original
            for (int i = 0; i < 200; i++)
            {
                sb.Append(Guid.NewGuid().ToString());
            }

            string messageSource = sb.ToString();

            Random rd = new Random();

            for (int i = 0; i < 100; i++)
            {

// Comment out following so that webSocketClient will send more data.
   //             int startPos = rd.Next(0, messageSource.Length - 2);
    //            int endPos = rd.Next(startPos + 1, messageSource.Length - 1);

                //             string message = messageSource.Substring(startPos, endPos - startPos);
                string message = messageSource;

                Console.WriteLine("Client:" + message);
                var data = Encoding.UTF8.GetBytes(message);
                webSocketClient.Send(data, 0, data.Length);

                if (!m_MessageReceiveEvent.WaitOne())
                    Assert.Fail("Cannot get response in time!");

                Assert.AreEqual(message, m_CurrentMessage);

Coordinator
Apr 1, 2012 at 7:16 AM

There is a configurable limitation for max allow request size:

maxCommandLength

http://supersocket.codeplex.com/wikipage?title=Basic%20configuration&referringTitle=Documentation

Apr 1, 2012 at 6:15 PM

Appreciate your reply.

How to deal with unknown size? In other words, client sends unknown size of serialized json object to server, and server needs to deserialize it? For example, a Json object with base64 image buffer.

Can you point me an example?

Thanks.

Coordinator
Apr 2, 2012 at 12:19 AM
Request size limitation is a must. You should send large message parts by parts.

Sent from my Windows Phone

From: htl2012
Sent: 4/2/2012 2:16 AM
To: kerry-jiang@hotmail.com
Subject: Re: How to send large amount of data [SuperWebSocket:350782]

From: htl2012

Appreciate your reply.

How to deal with unknown size? In other words, client sends unknown size of serialized json object to server, and server needs to deserialize it? For example, a Json object with base64 image buffer.

Can you point me an example?

Thanks.

Apr 5, 2012 at 8:08 AM

Still has 2 questions:

1. maxCommandLength is SuperWebSocket server configuration parameter. The client should get this parameter from server and then decide max command length sent to server? By the way, why default is 1024, which is very small?

2. After knowing the parameter, client will break big data set and sent them through a group of predefined commands (so that server knows start/end of the data). Server is able to assemble the pieces into a complete data set. Therefore, the server have to implement custom session to track the state of the session, so that it is able to assemble the complete data set sent from client. Is this the only way to make it work if using superwebsocket command?

3. I saw the SuperWebSocket has JsonSubCommand. What will happen if a json member is more than maxCommandLength?

Thanks.

Coordinator
Apr 5, 2012 at 8:17 AM
From: [email removed]
Sent: Thursday, April 05, 2012 4:08 PM
To: [email removed]
Subject: Re: How to send large amount of data [SuperWebSocket:350782]

From: htl2012

Still has 2 questions:

1. maxCommandLength is SuperWebSocket server configuration parameter. The client should get this parameter from server and then decide max command length sent to server? By the way, why default is 1024, which is very small?

The client shouldn’t get max command length from server. I think 1024 is enougth for message exchanging.

2. After knowing the parameter, client will break big data set and sent them through a group of predefined commands (so that server knows start/end of the data). Server is able to assemble the pieces into a complete data set. Therefore, the server have to implement custom session to track the state of the session, so that it is able to assemble the complete data set sent from client. Is this the only way to make it work if using superwebsocket command?

Yes, it is probably best practice if you want implement resumable large data transferring. BTW, you’d better save the parts of data to disk instead of in memory.

3. I saw the SuperWebSocket has JsonSubCommand. What will happen if a json member is more than maxCommandLength?

The connection will be dropped, and an error will be logged. So you should adjust the parameter according your application requirement.

Thanks.