Journey to Work (JTW) 2016

Datasets
1

This dataset contains a snapshot of commuting patterns for Census day 2016 (Tuesday 9th August). It is a complete enumeration via a self-completion Census form.

This is a companion discussion topic for the original entry at https://opendata.transport.nsw.gov.au/dataset/journey-work-jtw-2016
2

Hi.

I have downloaded the dataset JTW2016_csv.rar and unarchived it. However, two of the files failed. See messages below. Wondering if anyone else has experienced similar problems.

Thanks.

K.

Table 12 OSA2 by DSA2 by Mode15.csv (-1985867255 B)… Failed! (Attempted to read more data than was available)
Table 13 OTZ by DTZ by Mode15.csv (300135148 B)… Failed! (Wrong checksum)

3

Hi @k4werri, yea looks like there’s a problem with those two files. Looking into it and hopefully will get it resolved soon. Thanks

4

Thanks …

K.

5

Hi @k4werri, the problem is to do with the large file size of those two tables. We’re looking at spliting the files into more manageable sizes. Apologies about this, we’ll try to get it resolved as soon as possible. Thanks

6

Hi Alejandro
I too am having trouble with Table 13.

R has hit a memory limit and terminated part way through importing. It seems to me like the problem is that every field is coded which is what totally bloats the file size. The csv is a ridiculous 26GB!! R handles Table 12 without difficulty.

I’d be happy with uncoded data at the simplest level possible, with code frames and equivalences to aggregate from, say mode xx or from TZ up to SA2, SA3, SA4 etc.

Also, the single zipped download is huge for those of us not yet on NBN or corporate download speeds. I’m sure others would really appreciate the option of taking only the tables they wanted, or downloading them individually over time.
cheers
Gillian

7

More comments:

Table 12 won’t import into MS Access, too big.

Why does Table 12 have the additional geographic codes (SA3/4) for the destination SA2, but not for the origin SA2? And no O_SA2 code, just the name, which makes equivalencing to other sources of SA3/4 a bit painful.

8

@GPAK thanks for the feedback. We are currently in the process of pulling out from Table 13 non-essential text fields like SA Names. This will reduce the file size of Table 13 considerably.

Also, we be uploading individual .zip files for each table shortly.

Regarding Table 12, we recommend first processing the data in R (or other programming language) to extract only the required data for analysis.

We have attached SA2 codes to Table 12 and will be uploading in the coming days.

9

Hi, Any word on when Table 13 might be reduced in size? We are also having trouble taking that table into other applications due to the current 26gb size.

10

Hi @SCGIS_ARP, we just re-uploaded the files, hopefully they’re now more manageable. Thanks

11

Thanks …

K.

12

Hi,

I have downloaded the new files, unpacked them and loaded them into R. In particular, I’ve loaded table 13.

I had also, in the interim, managed to process the previously uploaded table 13 (fst format version which resulted in a very large file once unpacked).

There is a significant difference in the number of records between these two versions of the JTW 2016 table 13 data.

There must have been (i) some aggregation and/or (ii) some records present in the fst version that were not included in the newly uploaded version.

Are you able to provide some guidance on why the number of entries is different.

Thanks.

K.

13

Hi Kerrie,

Thanks for downloading and working with the revised data sets. You are correct that there is a change in number of records from the prior data sets. We observed that many small values ( less than .01 and essentially zero) were applied to hundreds (and sometimes millions) of records. We therefore removed records with counts less than .011. This was done to make file sizes more manageable while preserving data integrity.

You can find more information in the User Guide V1.0 that accompanies the 2016 JTW data tables.

Warm regards,
Paul

14

Hi Paul,

Thanks for your response.

Regards.

K.

15

Hi JTW team
Is there an estimated date for re-posting the revised data? I’m awaiting it to write up a report and an idea of timing would help me set a delivery time.
Thanks
Gillian

16

To GPAK
JTW16 - at this stage, you’re better off going to ABS direct. Their TableBuilder product will let you extract fine grained JTW data from 2016 Census. At workplace end, spatial coding is not too bad, as STA workplace destination zones used by ABS now are reasonably disaggregate.
Good luck
Tom

17

Thanks for the tip. I’ll look at ABS in the meantime.
Is there any update to when the BTS data release will be ready?

18

We would like to thank everyone for your interest in Journey to Work data.

Due to changes implemented by the ABS to confidentialise Census 2016 data and protect individuals from re-identification, TPA is not able to release JTW 2016 with TZ spatial variables. If you have downloaded JTW 2016 data with TZ spatial variables from this site, please delete it. TPA does not recommend it’s use. If you require further assistance or clarification, please email tpa@transport.nsw.gov.au.

JTW 2016 is available to download via the ABS website: https://auth.censusdata.abs.gov.au/webapi/jsf/login.xhtml. Please note, in releasing JTW 2016 the ABS has removed additivity and supressed all values smaller than 3.

Thank you.