1 Topic by mathmathou (edited by mathmathou 2016-10-19 05:11:20)

  • Registered: 2015-02-24
  • Posts: 446

Topic: [Solved] - HTTPGet - question for Dmitry

Hello Dmitry and all MVD fans,


As you may know, I've oriented my developments on systems aiming at scraping info from webpages. I use a lot of HTTPGet to retrieve web page source code and parse it looking for specific infos.


From time to time, I launch a "batch" check on pages (sometimes more than 20.000 URLs) and, when an error occurs, I get a DLGmessage from the system with the 404 error. The problem is that, when this occurs during a big loop (checking urls after urls), the system halts until I click on the OK button. And if I'm not behind the screen, we'll... the system waits for me.


My first question is : is there a way to avoid that message or, close it automatically after a certain amount of time for example ?


Second question : I assume you're using Indy HTTP. If this is correct, could you let us manage the exception ?
I' d like to do something like : 

try
     Source := HTTPGet(URL);
     <some parsing code>
Except
     On E: Exception do
          Begin
               Form1.Memo1.Lines.Add(E.Message);
          End;
End;

I hope I was clear enough for you to understand my question.


Wish you all a good day


Cheers



Mathias

I'm a very good housekeeper !
Each time I get a divorce, I keep the house
Zaza Gabor
  • Registered: 2014-02-14
  • Posts: 10,294

Re: [Solved] - HTTPGet - question for Dmitry

Hello.


Please download latest beta version here
https://www.dropbox.com/s/k1wd8mqe3qb35 … a.zip?dl=0


example:

try
     Source := HTTPGet(URL, True);
     <some parsing code>
Except
     Form1.Memo1.Lines.Add(ExceptionMessage);
End;
Dmitry.
  • Registered: 2015-02-24
  • Posts: 446

Re: [Solved] - HTTPGet - question for Dmitry

Woooo, nice move that Boolean switch in the HTTPGet and HTTPGetFile instruction !!


That does the trick perfectly and you can even personalize your message like smile

procedure Form1_Button4_OnClick (Sender: string; var Cancel: boolean);
begin
    try
        Source := HTTPGet('some url',True);
    except
        if ExceptionMessage <> 'HTTP/1.1 404 Not Found' then  Form1.Memo1.Lines.Add('Unknown Error') else Form1.Memo1.Lines.Add(ExceptionMessage);
    end;
end;

That's great Dmitry, thank you a million times!!


Two questions :
1- I hope you're not building betas just for me smile
2- Do you plan on adding that boolean switch to other functions like for example SQLExecute an so on ?


Thanks again a million


Cheers



Mathias

I'm a very good housekeeper !
Each time I get a divorce, I keep the house
Zaza Gabor
  • Registered: 2014-02-14
  • Posts: 10,294

Re: [Solved] - HTTPGet - question for Dmitry

1- I hope you're not building betas just for me

No )


2- Do you plan on adding that boolean switch to other functions like for example SQLExecute an so on ?

Please describe the situation when you need this addon.

Dmitry.
  • Registered: 2015-02-24
  • Posts: 446

Re: [Solved] - HTTPGet - question for Dmitry

Nothing specific at the moment really, just curiosity smile


Thanks again for the "except" thing

I'm a very good housekeeper !
Each time I get a divorce, I keep the house
Zaza Gabor