POST request issue with httr: desired table not retrieved

Question

Description: trying to retrieve historical data from Investing.com using httr library

Original page: https://www.investing.com/rates-bonds/austria-1-year-bond-yield-historical-data

Expected output: html table with historical data: sample table output

Script logic:

• Send a POST query with httr
• Prettify output of read_html method with html_table method

Issue:

• Script retrieves tables from the main page instead of the actual history table

Code:

library(httr)

url <- 'https://www.investing.com/instruments/HistoricalDataAjax'

# mimic XHR POST request implemented in the investing.com website
http_resp <- POST(url = url,
                 body = list(
                   curr_id = "23859", 
                   smlID = "202274", 
                   header = "Austria+1-Year+Bond+Yield+Historical+Data",
                   st_date = "08/01/2021", # MM/DD/YYYY format
                   end_date = "08/20/2021",
                   interval_sec = "Daily",
                   sort_col = "date",
                   sort_ord = "DESC",
                   action = "historical_data"
                 )
                )

# parse the returned XML
html_doc <- read_html(http_resp)
print(html_table(html_doc)[[1]])

You might notice that the URL used in the R script uses a different URL https://www.investing.com/instruments/HistoricalDataAjax compared to the original web-page https://www.investing.com/rates-bonds/austria-1-year-bond-yield-historical-data. The reason for this is apparently the link used in the POST request when setting the start and end dates. You may see this on the screenshot below:

XHR request header when setting the start and end dates

From what I see, when a user specifies a date for a particular security, website sends a query to HistoricalDataAjax with parameters and identifiers of securities/assets specified in the body of the request: Example of the requests's body after selecting dates

Answer 1

You can get the table in,

https://www.investing.com/rates-bonds/austria-1-year-bond-yield-historical-data

using rvest

library(rvest)
df = url %>%
  read_html() %>% 
  html_table()

df[[1]]
# A tibble: 25 x 6
   Date          Price   Open   High    Low `Change %`
   <chr>         <dbl>  <dbl>  <dbl>  <dbl> <chr>     
 1 Dec 09, 2021 -0.669 -0.672 -0.633 -0.695 11.69%    
 2 Dec 08, 2021 -0.599 -0.6   -0.549 -0.647 -2.28%    
 3 Dec 07, 2021 -0.613 -0.621 -0.536 -0.656 -7.54%    
 4 Dec 06, 2021 -0.663 -0.648 -0.565 -0.687 -0.30%    
 5 Dec 03, 2021 -0.665 -0.681 -0.577 -0.684 0.45%     
 6 Dec 02, 2021 -0.662 -0.59  -0.573 -0.669 0.46%     
 7 Dec 01, 2021 -0.659 -0.608 -0.577 -0.685 1.70%     
 8 Nov 30, 2021 -0.648 -0.697 -0.601 -0.736 -4.85%    
 9 Nov 29, 2021 -0.681 -0.715 -0.647 -0.745 -12.47%   
10 Nov 27, 2021 -0.778 -0.701 -0.701 -0.778 7.61%