Advice/Ideas on Sending Large Amounts of Data

[ktg8]

I began a project months ago to build a GPS tracking device and recently began working on it again. My rf266 takes gps data, parses it and saves it to an external memory chip. The gps will be tracking a slow moving object and may occasionally be out of range of the receiver (another rf266). I inserted a rpc to the gps node to ask every x minutes if the other node (bridged to the computer) is there. If it is, it takes the saved gps data out of the chip and sends it to Portal via rpc. At first I was sending each gps point individually, but most of the time the amount of rpc's trying to go out in a short period of time was too much and most of the data was lost. After reading another thread, I changed it so that each rpc held 4 gps points (after toying with it for a while, 4 was the max I could do before the packet size limit was exceeded). It sends out 10 or so packets before the rest is lost due to an overload of them trying to send out.

I have tried several things recently. One of them was that I tried to insert a delay (created one with a simple while loop) between rpc calls which didn't work (too small a delay did nothing and too large caused it to skip the whole process). I also tried to somehow incorporate a Hook event (if x = True send one through every second) but that didn't work for me either.

Because the object being tracked is a slow moving object, the devices may not be in communication for long enough to acquire lots of data that will need to be sent out and 40 points is just not enough.

I have very limited experience in programming and am sure there are many other things that I could try, not to mention perhaps the things that I have already tried could have worked but I did it wrong, but I cannot think of what to try next. So I was hoping that there may be someone out there that could potentially give me some ideas as to what to try next. Some other way to put some kind of a delay between rpcs so that they all make it?

Hope I was clear enough, had a hard time figuring out exactly how to ask my question.

Thank you for your time!

[gvoce]

Have you reviewed the GPS application note that we offer?

I am not sure I understand this statement .....

It sends out 10 or so packets before the rest is lost due to an overload of them trying to send out.

Are you saying the node sending out the GPS info stops? The app note might help you it discusses the large amount of GPS data and how to parse it.

[Can8DN]

Test posting (edited from Firefox and Chrome)

[Jheath]

Test posting (edited from Firefox and Chrome)

Test post too

[ktg8]

Been having issues trying to respond and going to give this another go with the different user name that tech support gave me. The following is the message that I have been trying to post.

Thanks for the reply!

Yes! I have taken a look at that document and it was extremely helpful when I was learning how to parse the GPS info.

I am going to go ahead and attach some of my code (which is what I should have done in the first place) to help show what my problem is. (I am currently away from home and only have a pdf of it)


The numCounter comes from the function that writes to the chip. It counts how many gps points were saved into the chip so that it knows how many to read off the chip when the info is sent to Portal. The num variable tells Portal how many points are in the rpc to pull out.

When the devices are within range, Portal sends an rpc to initiate this function to send the saved info.

The packets of info that I was talking about is the info in the rpc calls. It seems that I can send over 40 points successfully (10 rpcs) but after that, something gets clogged up and the rest of the info is never sent. I put a print statement in after the rpc call to make sure that it went through the loop as many times as I needed it to and also put in a hook "rpc sent" event to compare. It would complete the loop say 30 times, but only sent 10 times. So i need to come up with a way to delay between rpc calls or....??? Some way so that everything doesn't get clogged up and only sends some of the data and not all.

Hopefully I was bit clearer this time?

Thanks again!

[RPS]

ktg8Too

In Section 10 of the SNAP Reference manual (p120) it talks about buffers, pools and packet sizes. Long story short it’s possible to generated data faster than it can actually be sent out the radios.

And because the “transmit” functions are non-blocking (p87) it’s possible queue up data and backlog the buffers such that it appears to be stuck.

Real world this mean you can usually make 4 rpc calls before the buffers are all used up and you have to wait until the radio actually transmit something before you can put more data into the buffers via rpc calls. This is where the @setHook(HOOK_RPC_SENT) event hook comes in.

It gets called when the radio actually transmits a packet so another rpc call can enqueue data. One to prime the radio to start transmitting and in the @setHook(HOOK_RPC_SENT) you sent one until your run out of data. Because the HOOK_RPC_SENT will be called each time the radio transmits one of your packets. (Example script using @setHook(HOOK_RPC_SENT) is protoSleepCaster.py and is included with Portal.)

So back to your code, removed the “while I < numCounter” your generating and trying to send data faster than the radio can process the rpc calls. Factor out the stuff before the rpc call into a function because you are going to need it in two places now. Here and in the function defined below the @setHook(HOOK_RPC_SENT) event.

In the @setHook(HOOK_RPC_SENT) function decrement the global variable numCounter each time you rpc data back to Portal until its zero.

Review the protoSleepCaster.py sample to see the mechanics and tune the initial transmit 1-4 depending your HW. Difference RF Engines have different memory configuration and can support more rpc data buffering. Need to think event happens process data to clear the event. Not spin/poll to consume data. Very much like interrupt driven event processing.

You may want to search the forum for HOOK_RPC_SENT along with all the manuals now that you know what the actual problem is to find more examples and discussions.

Hopefully this helped.

RPS.

[ktg8]

Thank you very much! That explanation certainly helped! I never thought of using the hook_rpc_sent event, and that definitely makes sense. I will work on that this evening when I get home and give an update.

[ktg8]

Thank you!!! Works great!