Jump to content
Sign in to follow this  
Honest

How am I doing HTTP Web Requests wrong? (C# code but HTTP knowledge mi

Recommended Posts

Honest    12

My code:  http://ideone.com/lupFY5

 

I am trying to write a program which requests and prints this HTML:  http://ideone.com/LNf6zm

This is the HTML rendered when one visits http://www.entropedia.info/Chart.aspx?chart=Chart and clicks the (All) button under the table.

 

Currently, my program requests and prints this HTML:   http://ideone.com/P58zgm

This is the HTML rendered when one visits http://www.entropedia.info/Chart.aspx?chart=Chart and does not click any further links.

 

The easy way to tell the difference between the desired and obtained HTML is that the desired HTML is much longer.

 

I don't really know what Post Data I need to send.  LiveHTTPHeaders concatenated everything that happened to be in Chrome's Developer Tools -> Network -> Headers -> Form Data section when I analyze clicking the (All) button on the above Entropedia page, as is seen in Line 27 of my C# code.

 

If anyone can tell me what Line 27 should be, it would be GREATLY appreciated.  Thanks in advance for any help!

Share this post


Link to post
Share on other sites
»Paxman    4776

Omw to work will look in about 30 minutes.

 

Do you need to use C#? Cause T might be right about using curl, it's probably easier.

Share this post


Link to post
Share on other sites
Honest    12

Omw to work will look in about 30 minutes.

 

Whenever you get around it to is fine; it would be greatly appreciated.

 

You're probably both right about Curl being easier.  I will look into it as an option for the project.  I think I will face this particular problem in either language though.  It might be more of a "what POST data do I need to send" question than a language-specific question.

Share this post


Link to post
Share on other sites
»Paxman    4776

I am a wee bit backed up at work (everyone's getting fired lol), have you tried posting on stackoverflow? I've seen a lot of questions in my quick searches about http requests using c#.

Share this post


Link to post
Share on other sites
Honest    12

I am a wee bit backed up at work (everyone's getting fired lol), have you tried posting on stackoverflow? I've seen a lot of questions in my quick searches about http requests using c#.

 

No prob, if you get around to it another day, feel free to share any input.  I do have a lead at the moment; it seems the __VIEWSTATE components of the post data is causing at least one of the issues, since it changes each time the Web page is refreshed, etc.

Share this post


Link to post
Share on other sites
buster    9

I played around with the requests in Fiddler and it looks like the server validates the session cookie as well as the __EVENTVALIDATION before sending back the response (just my best guess). I shot off some requests with exact headers and __EVENTTARGET, __VIEWSTATE, and __EVENTVALIDATION parameters and got back the 'All' list. Removing the sessionId cookie header or any of those params returns an error or the default page.

 

If you are desperate, you can try sending a request to the page, parsing the above parameters from the response, then resend the request. I'm not 100% sure if the session cookie would go through the header correctly, but it's worth a shot.


There may be a way to return the markup through querystrings (would be easiest) but I haven't run across it.
 
Hopefully someone smarter than I can chime in with an actual solution.

 

Found a relevant StackOverflow link.

Share this post


Link to post
Share on other sites
Honest    12

Thank for the tip!

 

This is my current code:  http://ideone.com/CB6VsV

 

I'm getting the error HTML you mentioned now, as shown below  =(

 

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<head id="Head1"><title>
Entropedia: Error
</title><link rel="stylesheet" type="text/css" href="styles/default.css" /></head>
<body>
    <form name="form1" method="post" action="Error.aspx?chart=Chart" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPaA8FDzhkMDEwYjY4MGI5YTFmM2RWdl316frP7h2DHwW3e1y3KJhlng==" />
 
    <div>
 
<h1 id="ErrorTitle">Oops... something went wrong</h1>
<p id="ErrorText"> The error has been&nbsp;logged and will hopefully be fixed soon. You may try to use the other functions of the site until the error is fixed. We are sorry for any inconveniences. <strong>Please file a bug report</strong> in the <a href="/Page.aspx?page=Forum:Bugs">Bugs</A> section if the error persists, or contact the admin at support@entropedia.info</p>
 
    </div>
    </form>
</body>
</html>

Share this post


Link to post
Share on other sites
buster    9

Iirc you don't need the other context garbage after the __EVENTVALIDATION. I think the important params are the sessionId cookie, __EVENTTARGET, __VIEWSTATE, and __EVENTVALIDATION.

 

Just to compare... these are the headers I used and am getting a desired response back:

Host: www.entropedia.info
Content-Type: application/x-www-form-urlencoded
Cookie: ASP.NET_SessionId=abc123;

Content-Length: 1795
 

This is the request body (no linebreaks): 

__EVENTTARGET=abc123&__VIEWSTATE=abc123&__EVENTVALIDATION=abc123

 

Fill in the right values for the 4 params obv. I think sending the cookieId as a header instead of the actual cookie will help. Not sure if Content-Length matters but FIddler auto-appends it.

 

request.Headers.Add("Cookie", "ASP.NET_SessionId=abc123");

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×