I’ve been working with the robotevents API and have encountered a bizarre issue. Everything was going swimmingly until one day the API stopped responding. After some troubleshooting, myself and @BattleCh1cken determined that requests to robotevents.com work in curl and python, but hang in rust and C#, as well as an online tool like reqbin.com. I should add that it’s not an authorization issue as even simple requests to get the html of the home page hang. Any of the methods which don’t work for robotevents work fine when I put in a generic url like vexforum.com.
If anybody has a solution besides switching to python I’d be very grateful.
To elaborate:
This works:
import requests
import os
token = os.environ['TOKEN']
url = 'https://www.robotevents.com/api/v2/events'
headers = {
'accept': 'application/json',
'Authorization': 'Bearer '+ token
}
r = requests.get(url, headers=headers)
print(r.text)
But this times out:
use reqwest;
use std::env;
use std::error::Error;
use std::time::Duration;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let token = env::var("TOKEN").unwrap();
let client = reqwest::Client::new();
let request = client
.get("https://www.robotevents.com/api/v2/events")
.header("accept", "application/json")
.bearer_auth(token)
.timeout(Duration::from_secs(10))
.send()
.await?;
println!("{:#?}", request);
Ok(())
}
I’ve seen this happen as well. From what I can tell, RobotEvents is enforcing some kind of rate limiting. Generating a new token and using that for authentication usually fixes the issue for me
I’d rather not have to create a new token every time it stops working. I also shouldn’t need any token to get a simple html response from the home page.
Use a back-off on failed requests. Trap the error, wait, and retry. Here is my Python code:
def _get_robotevents_data(the_url):
df_retval = None
got_data = False
num_attempts = 0
max_attemps = 5
while num_attempts < max_attemps and got_data == False:
response = requests.get(the_url,headers=ROBOTEVENTS_HEADER)
json_results = json.loads(response.content.decode('utf-8',"ignore"))
try:
df_retval = pd.read_json(StringIO(json.dumps(json_results['data'])))
got_data = True
except:
num_attempts = num_attempts + 1
print("Error with response: " + json.dumps(json_results),flush=True)
time.sleep(30 + min(35,(num_attempts - 1 ) * 10))
if got_data:
while json_results['meta']['next_page_url'] is not None:
got_data = False
while num_attempts < max_attemps and got_data == False:
time.sleep(1)
try:
response = requests.get(json_results['meta']['next_page_url'],headers=ROBOTEVENTS_HEADER)
json_results = json.loads(response.content.decode('utf-8',"ignore"))
except:
json_results = dict()
if "data" in json_results.keys():
result_json = json.dumps(json_results['data'])
df_thispage = pd.read_json(StringIO(result_json))
df_retval = df_retval.append(df_thispage)
got_data = True
else:
print("No element 'data' in response: " + json.dumps(json_results),flush=True)
num_attempts = num_attempts + 1
if num_attempts >= max_attemps:
raise Exception("No element 'data' in response.")
else:
print("Waiting " + str(30 + min(35,(num_attempts - 1 ) * 10)) + " seconds for next pull request",flush=True)
time.sleep(30 + min(35,(num_attempts - 1 ) * 10))
else:
raise Exception("Unable to get data.")
return df_retval
It’s been solved! I had to add the header “accept-language: en” and add a user agent header. After some experimentation and testing, I found a bizarre behavior regarding the user agent value. I can only get a response from the website if the user agent either starts with “python-request” or contains “bot”. Otherwise no response will be sent regardless of how many times I retry. This has been corroborated in C#, Rust, and curl.
@jpearman any insight on this behavior?