Reading Undocumented Web APIs

Its often said that the internet is just APIs Application Programming Interfaces These are what allow applications to talk to one another and share data and functionality While many of the APIs are documented in great detail others are hidden behind a wall of silence This is likely the type of information that would be gamechanging or give you new unique functionality or efficiencies in a workflow But to get there its a whole other ballgame

This tutorial will equip you with the tools and knowledge you need to navigate the unknown and undocumented world of web APIs Well explore how to discover and interact with these APIs as well as the ethical and legal implications of doing so.

 Hunting the Untrackable

Lets say youre at a site with a search bar and some results but theres no API that automatically delivers the data This is an example of a hidden function an API that you didnt know about might be behind it This is how you find them out.

Dev Tools: in the Browser Your biggest advantage is your browsers own developer tools To see all the requests being made while youre using a webpage go to the Network tab You should be watching for XHR XMLHttpRequest or Fetch requests that are returning the data you need The last usually means an undocumented API

Exploratory Code Review: The answer can sometimes be found in a websites JavaScript code Just rightclick on any part of the page and click Inspect or Inspect Element Then look for API URLs or function calls that are used to manipulate data in the JavaScript code

WebSockets: More modern web applications could use WebSockets for realtime communication Tools like browser developer extensions will help you look at WebSocket traffic which might expose hidden API endpoints that are being sent in the messages

Discussion Forums and Code Repositories Outside of these methods developers frequently share their findings in online discussion forums and code repositories such as GitHub Searching these platforms with a keyword of the website youre interested in could result in some examples of undocumented API use

 Decoding the Signal

Once you spot a potential undocumented API you need to find out how it is working Here is what you need to do.

Analyze Formats of Request and Response Look at the format of the data that is transmitted between the website and the server For example a request might have parameters as search queries or filters and then the response can be in JSON XML or plain text

Experimentation Prepare test requests by mimicking the form of requests you see Tools like Postman could be very helpful to send those test requests and get the response from the server This gives you a clue on how the API operates as well as what the API expects in return

Document Everything: While you are testing it and learning write everything down It would be great if you could take the endpoints of the API the request parameters and the formats of the answer down The document you put together you will find is going to be very important in the future

 Laying the Foundation

Now that you have become familiar with the API jargon it is time to use it programmatically Here are a few methods

Programming Languages For instance Python is a programming language that has libraries for making HTTP requests and JavaScript can also work with it Using a library you can write code quickly to make requests from findings and parse responses

Scripting Tools: These scripting tools allow the user to use raw and customised HTTP requests with features like curl or Postman This is a good way to automate API interactions

Browser Extensions: You can take things a step further and create a browser extension where you intercept the requests to a website and insert your calls to the nonpublic API directly

 Ethical and Legal Aspects

Having said that when youre in the rush and stressed to use undocumented APIs you should consider the ethical and legal implications

Terms of Service: Most websites have a terms of service section It includes what is permissible to use APIs and what is not Using these undocumented APIs could violate the respective terms of service so be careful

Data Privacy: The presence of an undocumented API might not have the safety and security measures in place that a documented API has Be aware of what data you are fetching from these APIs and remember not to scrape or collect sensitive information without the right consent

Respecting Rate Limits: APIs have rate limits in place to avoid overburdening their servers So be nice to them and dont exceed the number of requests they can handle

Breaking Changes Undocumented APIs change freely without any concern Your code can also break because of some changes that are entirely out of your predictions Be ready to adjust and tend to your code

 Conclusion

While interacting with a web API is not different the major difference is that undocumented web APIs can be powerful The techniques in this book will help you unlock that power in a way that is ethical and respectful Remember do not go too far down the path Document everything and beware of the risks to your terms of service and the data privacy policy With the right approach you can turn undocumented APIs into a powerful tool on your development journey