TfNSW real-time map

Brag
1

TfNSW real-time map

1333×693 422 KB

So, it’s been a fairly exciting week for all data geeks alike! I’ve been working on a pretty simple real-time map, currently polled once a minute from once every 15 seconds from the TfNSW APIs.

At the moment, it only supports trains (that’s Sydney Trains and NSW TrainLink services). Once a few issues are resolved by the TfNSW Open Data team, I’ll include buses, light rail and ferries. Hopefully, I’ll also be able to increase the frequency of data polling. Trains, buses, light rail and ferries are now all on the map :slight_smile:

Stopping patterns for each service is also coming soon – pending the resolution of this issue.

Known issues:

  • NSW TrainLink services that are travelling within Sydney Trains’ network currently shown with duplicate markers. Logic is required to filter out NSW Trains data when it enters the metropolitan network since GPS data is delayed by ~5 mins and isn’t always accurate.
  • Hunter Line services are not labeled.
  • Clicking on marker shows the full extent of the route. It should be truncated to start and end station for the trip.

Fixed issues

  • Stopping patterns are missing in action pending resolution of this issue.

Feedback & suggestions are always appreciated :smiley:

Link: TfNSW real-time map

5 Likes
2

Morning,
Congratulations on your site, it looks great.

I am trying to get my own version up and running as well and I am wondering if I can ask you question about getting the data from the API. This is something that I have been trying to wrap my head around for a while.

When a client accesses your site, do you send that request through to the tfsnw API? So say 10 people access your site, is that 10 requests to your site and then 10 requests to tfnsw, or 10 requests to your site and then 1 request to tfsnw, with it those requests being server out of say a cache?

For example my idea is this.

  1. Have a background poller getting the data from the API and putting into a cache, say disk, every minute or so, already in the format I need to send to the client.
  2. All requests for my API, use that cache version.

This way I am not opening my server to manage a 1 to 1 relationship between the client and the tfnsw API and I think I will win overall as I have reduced the overall processing time as I am not proxying the client request.

Any thoughts on this approach would be appreciated.

3

This is fantastic. Any chance of sharing the github?

It seems very mature already - you are already detecting and reporting outages.

How are you making the coloured boxes with the initials in them? Its a design style I see everywhere, but I have no idea how it is typically done these days (or what it is called).

4

That’s basically what I have going! I have a cron job running once a minute to download the latest data and loading it onto a database. I’ll probably need to make it more sustainable as I’ve reached almost 2 million rows in just the past few days.

The map then calls on an endpoint on my own server that queries my own API which only looks up data on my own database.

This is definitely the best approach currently. I’ve been chatting with Yvonne and it looks like there are a few issues that the may need to be ironed out on their end before we can make more frequent requests. I had it set to download all modes of transport every half a minute (i.e. 5*2 = 10 per minute) and the servers weren’t too happy… but they’re working on it :slight_smile:

5

Hey Tom!

I do certainly plan to eventually. I’m not a huge fan of the current code (it’s rather cobbled together, as is all rapid prototyping code), so once I clean it up a bit… I’ll pop it up.

It just happened that the service has been hitting some “teething issues” over the past few days… which meant implementing the outage message was actually good for debugging! If the last two data updates have failed, it will show the TfNSW outage message.

It’s CSS :slight_smile: No idea what it’s called though… but TfNSW seems to love it.

.icon {
    color: white;
    background: #777;
    border-radius: 12px;
    border: 2px solid white;
    height: 24px;
    width: 24px;
    margin-left: -12px;
    margin-top: -12px;
    text-align: center;
    line-height: 22px;
    font-weight: bold;
}
.sydneytrains.icon.line-T3 {
    background: #f25323;
}
1 Like
6

Cheers Kenneth! Small world :slight_smile:

Lemme know when you push to github, super keen to see your stack xD

How is your memory / CPU load going with all the parsing?

7

How were you able to get the data in regards to what train is serving a particular route ie 8 cars A Set (Waratah)?

8

It’s part of the Trip ID (read the documentation here).

e.g. you’ll notice A.8 in the trip ID 175N.922.106.120-20160427.A.8.41509951 denotes that this is an A set with 8 carriages.

Similarly, H.8 in 282P.922.106.88-20160427.H.8.41507879 indicates this is an H set (Oscar) with 8 carriages.

edit: corrected to H set, not M set.

9

So, I busted the storage capabilities of my old server last week. Finally got around to fixing it tonight.

As a bonus, I’ve also added live bus and ferry data :slight_smile: Bus timetable is pending this.

<img src=“//cdck-file-uploads-global.s3.dualstack.us-west-2.amazonaws.com/business5/uploads/nsw/original/1X/192e3fdaf497746511f4d5c1ef4d0ebc140657b0.png” width=“690” height=“456”

1 Like
10

The data seems frozen as of around 5pm on 9th May.

11

Ah so that’s why I couldn’t find something I was after earlier.

12

This is awesome! Thanks for sharing.

I’m keen to see that too, even if it’s messy.

I tried to build pretty much this exact site for GovHack (I didn’t know about your version until now) but didn’t get far enough along with it.

I was building it on Mapbox GL JS rather than Leaflet though (client can load and show all route paths via vector tiles, the GTFS-R protobufs were downloaded directly from the client and decoded within the client JS application, and a custom basemap design to enhance the data display), with the goal to animate the markers so they look real time using GitHub - misterfresh/mapbox-animation: Animate a Geojson Source (my idea was to extrapolate out from the location given in the feed along the route polygon based on the speed and direction (forward/backward) to a maximum of the refresh interval using something like https://github.com/Turfjs/turf/tree/master/packages/turf-line-slice-along to de the extrapolate along a line, at least for buses and ferries. Trains since I think they are based on section midpoint might need to be treated slightly differently.

13

Thought I’d spend the Xmas/NY overhauling the map :slight_smile: The updated map can be found at anyt.rip/map

Changelog

  • Refresh rate is now ~15-30s for all modes except buses. Buses is ~1-2 mins due to large dataset.
  • Map should now correctly show the stopping pattern of added trips and runs with transpositions
  • Marker movement is animated
  • Ability to select which modes to display (buses hidden by default to reduce lag).
  • Click on a vehicle to highlight a particular trip and get additional information
  • Now shows route and stop locations
  • Vehicle direction arrow for all modes with bearing data (except Sydney Trains).

2242×1718 1.34 MB

1 Like
14

Looking great @jxeeno. Can you update your logos of the services to the official ones? Including the ‘burnt’ orange for the T for regional trains (rather than the non approved purple R?)

1 Like
15

Will do @yvonne.lee.

Is the burnt orange roundel OK for describing regional services (coach and trains) collectively?

16

Hmmm I think so. Collectively I think it’s known as NSW Trainlink but the mode icons should be the Coach (purple) and Train (Burnt Orange).

I grabbed the below from our brand guidelines. Note you can also use the pictograms. Usually the letters are used for Wayfinding but has become the common way for app developers to indicate the modes and is recognisable by our customers.


1 Like
17

Great work! would be cool to see a refresh timer next to “last updated” to give the user a better picture on when the next data set will be loaded

18

Thanks for the suggestion, @PeterL2F! I’ve now updated the “last updated” list to include a timer:

1 Like
19

Nice! It looks very slick.

I remember you mentioned in another post that you are putting some documents together. Will they be publicly available?

20

Thanks :slight_smile: I’ll look into releasing a public version. I currently has a fair amount of internal stuff in it as well.