What do people do for server infrastructure?


#1

Hi,

I am curious about how people deal with the need to have a server to provide realtime updates to apps. When I first started to look at developing realtime support for my applications, I was hoping that I would be able to do queries to the feed targeted at particular stops. As this is not the case, I set about developing the apps to use my server.

I set the server up to download from the feed once per minute from 05:00 to 23:59. I was somewhat horrified to find that this amounted to about 5GB per day, which used up half of my monthly broadband allocation in five days! As this is obviously not tenable, I was wondering what other people do.

Thanks for any insight anyone can offer.

Regards,
Ned


#2

Hi Ned

Will let others respond to you re the realtime updates. FYI we are intending to offer some parameters via an API but for now it is the full feed. (No ETA for this either that we can share.)

Some of the app developers who are on this forum should be able to share how they do things.

Regards
Yvonne


#3

Hi Yvonne,

Thanks for the info. I’ll look forward with interest for any developments.

Regards,
Ned


#4

Hi Ned, Do you Accept-Encoding: gzip when you pull the feed?


#5

Hi Jayen,

I didn’t know anything about that. So this (presumably with a “–header” on the curl command) will cause the feed to be compressed? That should certainly help!

Thanks,
Ned


#6

If you’re using curl, use --compressed. Just setting the header isn’t much use if you can’t understand the response (marked with Content-Encoding: gzip).


#7

Ned

I don’t think there’s much you can do in terms of the high level of data transfer. @jayen’s suggestion of ensuring that you’re allowing the server to compress the content is a good one but, beyond that, this is a series of non-incremental GTFS RT feeds. They’re updated every minute or so, so you can’t build server infrastructure without pulling down all the feeds every time they’re updated!

One other option might be to use a third party Stop Monitoring / Data API so you can do the ‘per stop’ requests that you’re after – drop me a private message or email, as we’re launching something similar in the very near future and looking for beta testers.

Best

Carl