HTTP client error 403

Question

I have the following problem: when I click a button on a site (http://domain-location.com) I receive information back to me and the url is changed to http://domain-location.com?key=value. I want to create a sample http client to receive information easily. Here is my code:

function DoRequest(aVal: string): string;
 const DOMAIN = 'http://domain-location.com';
 var
   request: TIdHTTP;
   responseStream: TMemoryStream;
   responseLoader: TStringList;
   urlRequest: string;
 begin
   request := TIdHTTP.Create(nil);
   responseStream := TMemoryStream.Create;
   responseLoader := TStringList.Create;

   try
     try
       // accept ranges didn't help
       // request.Response.AcceptRanges := 'text/json';
       // request.Response.AcceptRanges := 'text/xml';
       urlRequest := DOMAIN + '?key=' + aVal;
       request.Get(urlRequest, responseStream);
       responseStream.Position := 0;
       responseLoader.LoadFromStream(responseStream);
       Result := responseLoader.Text;
     except on E: Exception do
       Result := e.Message;
     end;
   finally
     responseLoader.Free;
     responseStream.Free;
     request.Free;
   end;
 end;

EDIT After the first answer I edited my function (still not working):

 function DoRequest(aVal: string): string;
 const DOMAIN = 'http://domain-location.com';
 var
   request: TIdHTTP;
   responseStream: TMemoryStream;
   responseLoader: TStringList;
   urlRequest: string;
   uri: TIdURI;
 begin
   request := TIdHTTP.Create(nil);
   responseStream := TMemoryStream.Create;
   responseLoader := TStringList.Create;
   request.CookieManager := TIdCookieManager.Create(request);
   uri := TIdURI.Create(DOMAIN);

   try
     try
       // accept ranges didn't help
       // request.Response.AcceptRanges := 'text/json';
       // request.Response.AcceptRanges := 'text/xml';
       urlRequest := DOMAIN + '?key=' + aVal;
       request.CookieManager.AddServerCookie('cookie1', uri);
       request.CookieManager.AddServerCookie('cookie2', uri);
       request.CookieManager.AddServerCookie('cookie3', uri);
       request.Get(urlRequest, responseStream);
       responseStream.Position := 0;
       responseLoader.LoadFromStream(responseStream);
       Result := responseLoader.Text;
     except on E: Exception do
       Result := e.Message;
     end;
   finally
     responseLoader.Free;
     responseStream.Free;
     request.Free;
   end;
 end;

And after I do the request the result is: HTTP1.1 403 Forbidden. I inspected the page and the button I click is in a form like this:

<form action="http:/domiain.com" method="GET">
   <input type="text" name="key">
   <input type="submit" value="Click">
</form>

When I type http://domain-location.com?key=value there is no problem. Any idea how to fix it?

Answer 1

The problem was with the UserAgent:

function DoRequest(aVal: string): string;
  const DOMAIN = 'http://domain-location.com';
var
  request: TIdHTTP;
  urlRequest: string;
begin
  request := TIdHTTP.Create(nil);

  try
    try
      request.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36';
      urlRequest := DOMAIN + '?key=' + aVal;
      Result := request.Get(urlRequest);
    except on E: Exception do
      Result := e.Message;
    end;
  finally
    request.Free;
  end;
end;

Answer 2

Check the actual request that is sent by your browser. 403 suggests that there is some sort of authentication going on. This might be a token or a cookie that your browser has, and sends with the request, but your sample client application may not have. Open the browser debugging panel to check the get request made by your browser, and compare it to the one made by your application. I bet there will be some difference.

Answer 3

If cookies are involved, then you should GET the original HTML page first so the server can send whatever cookies it needs to be posted back when the button is "clicked", then you can GET the next page and let TIdHTTP post whatever cookies it had received.

Try this:

function DoRequest(const aVal: string): string;
const
  DOMAIN = 'http://domain-location.com';
var
  request: TIdHTTP;
begin
  try
    request := TIdHTTP.Create(nil);
    try
      request.Get(DOMAIN, TStream(nil)); // get cookies, discard HTML
      Result := request.Get(DOMAIN + '?key=' + TIdURI.ParamsEncode(aVal));
    finally
      request.Free;
    end;
  except
    on E: Exception do
      Result := e.Message;
  end;
end;