StatusCode 401 Error in GTFS Realtime Feeds

1

Hi, I have got some Node JS code to fetch GTFS realtime vehicle position feeds from ferries, trains, buses and light rails. It is working fine but sometimes I get this seemingly random “statusCode”:401 errors in the returned response, but next time the returned results fixes itself up and become OK. I fetch the realtime feeds every 15 seconds. Is it because the authentication takes too long to respond?

An example of such error in JSON is:

{"statusCode":401,"body":{"0":123,"1":32,"2":34,"3":69,"4":114,"5":114,"6":111,"7":114,"8":68,"9":101,"10":116,"11":97,"12":105,"13":108,"14":115,"15":34,"16":58,"17":32,"18":123,"19":32,"20":34,"21":84,"22":114,"23":97,"24":110,"25":115,"26":97,"27":99,"28":116,"29":105,"30":111,"31":110,"32":73,"33":100,"34":34,"35":58,"36":34,"37":48,"38":48,"39":48,"40":48,"41":48,"42":49,"43":53,"44":53,"45":52,"46":54,"47":49,"48":102,"49":57,"50":51,"51":49,"52":53,"53":45,"54":51,"55":98,"56":49,"57":53,"58":48,"59":98,"60":52,"61":34,"62":44,"63":32,"64":34,"65":69,"66":114,"67":114,"68":111,"69":114,"70":68,"71":97,"72":116,"73":101,"74":84,"75":105,"76":109,"77":101,"78":34,"79":58,"80":34,"81":50,"82":48,"83":49,"84":54,"85":45,"86":49,"87":49,"88":45,"89":48,"90":50,"91":84,"92":49,"93":53,"94":58,"95":51,"96":57,"97":58,"98":52,"99":57,"100":46,"101":48,"102":52,"103":48,"104":43,"105":49,"106":49,"107":58,"108":48,"109":48,"110":34,"111":44,"112":32,"113":34,"114":77,"115":101,"116":115,"117":115,"118":97,"119":103,"120":101,"121":34,"122":58,"123":34,"124":67,"125":97,"126":108,"127":108,"128":105,"129":110,"130":103,"131":32,"132":97,"133":112,"134":112,"135":108,"136":105,"137":99,"138":97,"139":116,"140":105,"141":111,"142":110,"143":32,"144":105,"145":115,"146":32,"147":110,"148":111,"149":116,"150":32,"151":97,"152":117,"153":116,"154":104,"155":101,"156":110,"157":116,"158":105,"159":99,"160":97,"161":116,"162":101,"163":100,"164":34,"165":44,"166":32,"167":34,"168":82,"169":101,"170":113,"171":117,"172":101,"173":115,"174":116,"175":101,"176":100,"177":85,"178":114,"179":108,"180":34,"181":58,"182":34,"183":47,"184":118,"185":49,"186":47,"187":103,"188":116,"189":102,"190":115,"191":47,"192":118,"193":101,"194":104,"195":105,"196":99,"197":108,"198":101,"199":112,"200":111,"201":115,"202":47,"203":98,"204":117,"205":115,"206":101,"207":115,"208":34,"209":44,"210":32,"211":34,"212":82,"213":101,"214":113,"215":117,"216":101,"217":115,"218":116,"219":101,"220":100,"221":77,"222":101,"223":116,"224":104,"225":111,"226":100,"227":34,"228":58,"229":34,"230":71,"231":69,"232":84,"233":34,"234":32,"235":125,"236":32,"237":125},"headers":{"cache-control":"no-cache=\"set-cookie\"","content-type":"application/json","date":"Wed, 02 Nov 2016 04:39:48 GMT","server":"Apache-Coyote/1.1","set-cookie":["AWSELB=777BE349184966AA129955C33D987E6DD8D703B5EF98646B5F5BFCF6E8BE2BEE935448000E243FA1120100A38CFDC57F4F47F05B5BDB933678B8A8E1D7005B3AB82BA260F3;PATH=/"],"content-length":"238","connection":"Close"},"request":{"uri":{"protocol":"https:","slashes":true,"auth":null,"host":"api.transport.nsw.gov.au","port":443,"hostname":"api.transport.nsw.gov.au","hash":null,"search":null,"query":null,"pathname":"/v1/gtfs/vehiclepos/buses","path":"/v1/gtfs/vehiclepos/buses","href":"https://api.transport.nsw.gov.au/v1/gtfs/vehiclepos/buses"},"method":"GET","headers":{"$ref":"$[\"socket\"][\"_httpMessage\"][\"agent\"][\"options\"][\"headers\"]"}}}
2

By the sounds of it you might be being rate limited, what happens if you change the interval to every 60 seconds - do all those requests go through fine?

Actually it might not be that since https://opendata.transport.nsw.gov.au/site/en_us/gs-api-basics.html mentions receiving a 403 error for that.

3

Hi @hkotfnsw, are you following the code sample that we provide in the api explorer? I’ve tried it both from the API explorer and command line using cURL and haven’t had any problems. Please make sure you follow the steps provided, only saying this because it’s a 401 (authentication) error. Feel free to private message me the code snippet you’re using so I can take a quick look at it.

Thanks,
Alex

4

Thanks for your replies. @phiali, it seems the problem is caused by the throttle limit. Instead of sending out my 5 requests (buses, Sydney trains, NSW trains, ferries and light rail) at the same time, I changed to sending out the requests one second after each other - and I no longer get the 401 authentication error.

The API basic help page seems to suggest that I need a “bronze” level API/application so I can have 5 requests per second. How can I check, and how I can I obtain bronze level? Also, if the 401 error is caused by throttling, shouldn’t I expect a 403 error (instead of 401)?

Any help is appreciated.

5

Hi @hkotfnsw

The default level is Bronze. You should be fine with that given your needs and looks like we need to update our documentation as it is a 401 you get if you are over the limit. It does have this:

Note: You can also receive HTTP 401 status codes if you go over your limits. See Limits for details.

I think the reference to 403 is when you exceed your daily limit.

But we have also had a few reports and have raised an issue about ‘random’ 401 errors that are happening. We’re trying to get more verbose logging to investigate this issue. I’m glad to hear your solution about the requests sequentially has helped.

6

Hi @yvonne.lee,

We also see occasional ‘random’ 401 errors when accessing some of the realtime GTFS bundles, but these are generally resolved by retrying the request. Do you have a link to a ticket or post regarding these, so that I can keep an eye on the issue? Also, please let me know if you would like any information about the requests we make (times, etc) which may help you to diagnose the issue.

Thanks,

David

7

thanks @david.else. Keep an eye on this post and we’ll update it here.

As for logs etc that may come in handy. I’ll put a request out if we need more info. Thanks!

8

Hi @yvonne.lee, just following up see if there is any news on this 401 status error? I can still see this error coming up intermittently. Thanks.

9

Hey guys

There is a fix but we’re scheduling it to be pushed into the production environment. We’ll advise when this happens (should be the next couple of days).

Thanks
Yvonne

10

hey guys, this fix should be in now (in today’s scheduled maintenance).

Can you all keep an eye on things and if there are continued issues please advise us.

Thanks!

11

Hi @yvonne.lee, FYI, I am still seeing this 401 error occasionally, but unfortunately hard to observe (e.g. if it is happening more or less)… will continue keep an eye out for this. Cheers.

12

thanks for advising us @hkotfnsw. We’ll advise our vendors and investigate further.

13

Hi @yvonne.lee, I am just following up to see if this issue has/will be addressed? Would you know what the limits are for hitting the feeds? Is it per-account, or per API key? The reasons is because in my project, we are hitting the vehicle position and realtime feeds for all 5 transport modes, at the same time. So that’s 10 requests. Then we will have our development, test and production environments, meaning 30 requests at the same time.

p.s. currently for the 10 requests, I am leaving a one second gap between the requests so that the vehicle position and realtime bus feed requests fire away first, then in the next second the Sydney Trains vehicle position and realtime bus feed requests fire away, then the next second with NSW Trains… etc. This has reduced the 401 error quite a bit, but it is still there.

14

Hi there

A fix was put in place but it didn’t seem to actually fix the problem. The vendors are still looking into it though… and I have no ETA.

At this stage the limits are by Account. We are soon moving to API key limits. Note for the GTFS-r there are ‘global’ limits which is what we suspected is causing the 401s.

Can you PM me the email address you use for the account and I can review what level you are set at and adjust appropriately. The 1 second gap is a good idea btw.

Thanks
Yvonne