For October 2021, we prioritised some major enhancements.
As the demand for OdataLink continued to grow in leaps and bounds, we decided it was time to have a look back at different systems in place to ensure they would scale.
- Consolidation OData Feeds
- Enhanced Logs
- Enhanced Preview of Data within Power BI and Excel
Consolidation OData Feeds
The addition of Consolidation OData Feeds is without a doubt the most exciting enhancement added to OdataLink this year.
It allows you to download data for all your data files using a single OData feed URL. It does so by paging and chaining the requests for all your data files together.
No longer will you need to download data for individual data files and append them together.
Instead, you can pull the data from the Consolidation OData Feeds and have OdataLink do all the hard work for you.
It is currently available in BETA form while we fine tune and iron out all the bugs.
New DataFiles Endpoint
To make sense of the data, all records include a new DataFile field. This field can also be used in conjunction with the DataFiles endpoint.
Enhanced Logs
After auditing the logs, we found that this part of our app was actually quite heavy and cumbersome.
The amount information stored in the logs was actually quite large. In contrast, the core data that manages the accounts, models, data files was actually quite small.
To reduce the load on our servers, we have reengineered the logging mechanism to be more scalable, efficient and transparent.
- First, we moved the logs data to a separate database.
- Secondly, we reengineered the structure of the the logs to decouple the requests received by OdataLink from those made to your data file.
- Thirdly, we’ve also devised a way to group associated requests together.
Ultimately, this ensures that logging requests will not be a bottle-neck when providing the data.
Checking out the logs using the updated user interface
From a visual’s point of view, we updated the log screen to be leaner and list less information at once.
With the latest update, requests are grouped with the initial request. This reduces the amount of information displayed on the log screen. For further details, you can drilldown on a specific item to see the individual requests.
What’s next for the logs
In the future, we will enhanced the log screen to show additional information (ip address, response codes from your data file, etc.) as well as provide a way for you to notify us of any issues that should be investigated (e.g. when a request fails).
Enhanced Preview of Data within Power BI and Excel
One of the most recurring questions we receive related to Error 1006 that appears in logs.
This error exist because MYOB and Xero only support a certain number of records per page. Power BI and Excel, on the other hand, always try to preview the data in pages of 1000 records.
With the latest update, OdataLink handles these requests differently and will no longer raise an error.
For all initial request received, OdataLink assigns a session id which gets included in all future calls. It also caches the initial $Top parameter received asking for 1000 records.
For all subsequent requests, we lookup the initial request and identify if we reached the cap of 1000 records. Once we do, we simply stop including the odata.nextlink property. This essentially informs Power Query that all records have been retrieved.
Au revoir Error 1006 (pun for those French speakers, au revoir means both goodbye and see you soon which in this case, we won’t).