Swagger API schema has multiple errors


#1

I notice that inside of the “API Explorer” there’s a backend which describes all of the API Endpoints, and presents Swagger (OpenAPI) 2.0 endpoints. It looks like the underlying project is Swagger UI, with some custom backend bits written by the contractors who set this service up.

I would really like a thing I can easily hit that contains all the endpoints described with valid Swagger schemas. At present, I am emulating a bunch of calls that would normally be executed by the Javascript of Swagger UI.

After some hacking around, I’ve got something working which enumerates these endpoints from the Catalog, grabs the Application Keys, and dumps out the Swagger schemas. That’s great, but I notice some issues with the Swagger schemas. I’ve been mainly hitting the Realtime endpoints, but I suspect these issues are global.

I’m still in the process of packaging up what I’ve got, but it would be made much easier if the issues below were fixed:

Issue #1: In the Swagger description for the Vehicle Position API, there is a Response model defined, which has a type of object. I don’t think this is valid according to the spec, and indeed when I load it into an API generator for Swagger, and try to make a request, it returns an empty object because it’s unable to parse the protobuf-formatted GTFS-realtime data.

I managed to fix this in the Swagger files by tweaking the endpoints response type for 200 from:

"200" : {
   "description" : "The request has been processed successfully.",
   "schema" : {
      "$ref" : "#/definitions/Response"
   },
   "examples" : {}
},

to:

"200" : {
   "description" : "The request has been processed successfully.",
   "schema" : {
      "type" : "file"
   },
   "examples" : {}
},

After this, the Response definition can be removed from the schema.

Issue #2: The endpoints themselves have nonsense operationIds.

"/sydneytrains" : {
   "get" : {
      "operationId" : "sydneytrains_res4_method0",
      /* ... */
   }
}

Issue #3: The basePath attribute at the root of the Swagger schema has a stray slash at the end. This normally wouldn’t be a problem, but the extra slash causes TfNSW’s webserver to be sad:

Traceback (most recent call last):

swagger_client.rest.ApiException: (404)
Reason: Not Found
HTTP response body: {
	"ErrorDetails":
	{
		"TransactionId":"XXXXXXXXXXXXXXXXXXX",
		"ErrorDateTime":"2016-04-24T00:00:00.000+10:00",
		"Message":"Requested resource was not found",
		"RequestedUrl":"/v1/gtfs/vehiclepos//sydneytrains",
		"RequestedMethod":"GET"
	}
}

Once I hack up the schema a bit, I can call everything via the generated API. But it’s a bit ugly looking:

$ java -jar swagger-codegen-cli.jar generate -i realtime.json -l python -o py
$ cd py
$ python
import swagger_client
client = swagger_client.ApiClient(header_name='Authorization', header_value='Bearer xxxxx-xxxxxx-xxxxxxx-xxxxxxxxxx')

# Fix issue #3 by trimming the / from the end
client.host = client.host[:-1]

# Setup the SydneytrainsApi
syd_trains = swagger_client.SydneytrainsApi(client)

# Find all the trains! With this one weird method name (Issue #2)
res = syd_trains.sydneytrains_res4_method0()

# Print out the path to the protobuf blob
print res

I can then read that protobuf blob successfully.


#2

I’ve analysed the other APIs for these issues, and published my code for getting the Swagger schemas. This should allow others to use Swagger’s code generation to make API libraries for many languages – however it is subject to the caveats noted in this thread.

If the Swagger was all fixed up, at this point there would be no requirement for each person to build their own API client, and everyone can use code generation to build language bindings.

It looks like that everything suffers from the response type specified in Issue #1, even the data which is supposed to return JSON (such as the Road Hazards feed). Looking at the Swagger issue tracker, they struggle to represent GeoJSON data. As a result I think every API should return "type": "file" for now.

All methods in all APIs have nonsense operationIds (Issue #2 above), and have a trailing res\d+_method0:

$ ack operationId
v1_publictransport_facilitiesandoperators.json
54:        "operationId": "GetLocationfacilities_res0_method0"

v1_gtfs_alerts.json
54:        "operationId": "SydneyTrains_res4_method0" <-- Should be GetSydneyTrains
104:        "operationId": "GetBuses_res0_method0"
154:        "operationId": "GetNSWTrains_res3_method0"
204:        "operationId": "GetLightRail_res2_method0"
254:        "operationId": "GetFerries_res1_method0"

v1_gtfs_realtime.json
54:        "operationId": "SydneyTrains_res4_method0"
104:        "operationId": "Buses_res0_method0"
154:        "operationId": "NSWTrains_res3_method0"
204:        "operationId": "LightRail_res2_method0"
254:        "operationId": "Ferries_res1_method0"

v1_gtfs_vehiclepos.json
54:        "operationId": "sydneytrains_res4_method0" <-- Inconsistent capitalisation
104:        "operationId": "Buses_res0_method0"
154:        "operationId": "NSWTrains_res3_method0"
204:        "operationId": "LightRail_res2_method0"
254:        "operationId": "Ferries_res1_method0"

v1_publictransport_timetables_complete_gtfs.json
54:        "operationId": "GetSydpubtransgtfs_res0_method0"

v1_publictransport_timetables_complete_transxchange.json
54:        "operationId": "GetTransxchange_res0_method0"

v1_gtfs_schedule.json
54:        "operationId": "Trains_res4_method0" <-- Should be SydneyTrains
104:        "operationId": "Buses_res0_method0"
154:        "operationId": "NSWTrains_res3_method0"
204:        "operationId": "LightRail_res2_method0"
254:        "operationId": "Ferries_res1_method0"

v1_ttds.json
54:        "operationId": "GetTTDSEvents_res0_method0"
141:        "operationId": "GetTTDSRoute_res2_method0"
228:        "operationId": "GetTTDSProgress_res1_method0"

v1_live_cameras.json
54:        "operationId": "GetTrafficCamera_res0_method0"

v1_live_hazards.json
54:        "operationId": "GetClosedTrafficIncidents_res10_method0"
104:        "operationId": "GetClosedFlood_res7_method0"
154:        "operationId": "GetOpenRoadwork_res17_method0"
204:        "operationId": "GetAllMajorEvent_res12_method0"
254:        "operationId": "GetOpenAlpine_res2_method0"
304:        "operationId": "GetOpenFlood_res8_method0"
354:        "operationId": "GetAllTrafficIncidents_res9_method0"
404:        "operationId": "GetAllFlood_res6_method0"
454:        "operationId": "GetAllAlpine_res0_method0"
504:        "operationId": "GetAllFire_res3_method0"
554:        "operationId": "GetClosedRoadwork_res16_method0"
604:        "operationId": "GetOpenTrafficIncidents_res11_method0"
654:        "operationId": "GetClosedMajorEvent_res13_method0"
704:        "operationId": "GetAllRoadwork_res15_method0"
754:        "operationId": "GetClosedAlpine_res1_method0"
804:        "operationId": "GetOpenFire_res5_method0"
854:        "operationId": "GetOpenMajorEvent_res14_method0"
904:        "operationId": "GetClosedFire_res4_method0"

v1_live_status.json
54:        "operationId": "GetTrafficSiteStatus_res0_method0"

v1_roads_static_loadingzones.json
54:        "operationId": "GetLoadingzone_res0_method0"

v1_roads_static_offstreetparking.json
54:        "operationId": "GetOffstreetparking_res0_method0"

These APIs have incorrect basePath (Issue #3 above), and have an incorrect trailing slash:

  • Public Transport - Facilities and Operators
  • Public Transport - Realtime - Alerts
  • Public Transport - Realtime - Vehicle Positions
  • Public Transport - Timetables - Complete - GTFS
  • Public Transport - Timetables - Complete - TransXChange
  • Public Transport - Timetables - For Realtime
  • Roads - Realtime
  • Roads - Realtime - Cameras
  • Roads - Static - Loading Zones - Kerbside
  • Roads - Static - Parking - Off-Street

The remaining APIs do not have this error.


#3

I’ve worked around these issues by writing another script which refactors the API.

This lets you work with the API as one package, which makes it much easier to work with code generation. I’m not really happy with the way that OAuth2 works in it just yet.

$ python -m monorail.tools.swaggify -t monorail/base_tfnsw_api.json -o apis/tfnsw_api.json apis/v1_*.json
$ java -jar swagger-codegen-cli.jar generate -i apis/tfnsw_api.json -o apis/tfnsw_api -l python

$ cd apis/tfnsw_api
$ python
>>> import swagger_client
>>> dir(swagger_client)
['AlpineApi', 'ApiClient', 'BusesApi', 'CamerasApi', 'Configuration',
'Error', 'EventsApi', 'FacilitiesandoperatorsApi', 'FerriesApi', 'FireApi',
'FloodApi', 'GtfsApi', 'IncidentApi', 'LightrailApi', 'LoadingzonesApi',
'MajoreventApi', 'NswtrainsApi', 'OffstreetparkingApi', 'ProgressApi',
'RoadworkApi', 'RouteApi', 'StatusApi', 'SydneytrainsApi', 'TransxchangeApi',
'__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__',
'absolute_import', 'api_client', 'apis', 'configuration', 'models', 'rest']
>>> swagger_client.Configuration().auth_settings = lambda: {'oauth2': {'in': 'header', 'key': 'Authorization', 'value': 'Bearer xxxxxxx'}}
>>> syd_trains = swagger_client.SydneytrainsApi()
>>> syd_trains.vehicle_positions()
'/tmp/tmpFOO'

I can also grab the static GTFS data with this:

>>> gtfs = swagger_client.GtfsApi()
>>> gtfs.sydpubtransgtfs()
'/tmp/tmpBAR'
$ unzip -t /tmp/tmpBAR
Archive:  /tmp/tmpBAR
    testing: agency.txt               OK
    testing: stops.txt                OK
    testing: routes.txt               OK
    testing: calendar.txt             OK
    testing: calendar_dates.txt       OK
    testing: shapes.txt               OK
    testing: trips.txt                OK
    testing: stop_times.txt           OK
No errors detected in compressed data of /tmp/tmpBAR.

I have made a pre-built Swagger API schema available.

The POST methods for routing information I haven’t tested yet.


#4

Thanks and noted micolous.
I’ll raise it tomorrow and see what we can do to get it updated.

Appreciate the analysis and info!

Regards
Yvonne