Figure out if a website has restricted/password protected area -

-

i have big list of websites , need know if have areas password protected.

i thinking doing this: downloading of them httrack , writing script looks keywords "log in" , "401 forbidden". problem these websites different/some static , dynamic (html, cgi, php,java-applets...) , of them won't use same keywords...

do have better ideas?

thanks lot!

looking password fields far, won't sites use http authentication. looking 401s http authentication, won't sites don't use it, or ones don't return 401. looking links "log in" or "username" fields more.

i don't think you'll able entirely automatically , sure you're detecting password-protected areas.

you'll want take library @ web automation, , write little program reads list of target sites file, checks each one, , writes 1 file of "these passworded" , "these not", , might want go manually check ones not, , make modifications program accomodate. using httrack great grabbing data, it's not going detection -- if write own "check password protected area" program general purpose hll, can more checks, , can avoid generating more requests per site necessary determine password-protected area exists.

you may need ignore robots.txt

i recommend using the python port of perls mechanize, or whatever nice web automation library preferred language has. modern languages have nice library opening , searching through web pages, , looking @ http headers.

if not capable of writing yourself, you're going have rather difficult time using httrack or wget or similar , searching through responses.


Comments

Post a Comment

Popular posts from this blog

c++ - How do I get a multi line tooltip in MFC -

-
right now, have tool tip pops when hover on edit box. problem tool tip contains multiple error messages , in 1 long line. need have each error message on own line. error messages contained in cstring new line seperating them. my existing code below. bool ontooltiptext(uint, nmhdr* pnmhdr, lresult* presult) { assert(pnmhdr->code == ttn_needtexta || pnmhdr->code == ttn_needtextw); // need handle both ansi , unicode versions of message tooltiptexta* pttta = (tooltiptexta*)pnmhdr; tooltiptextw* ptttw = (tooltiptextw*)pnmhdr; // tchar szfulltext[256]; cstring strtiptext=_t(""); uint nid = pnmhdr->idfrom; if (pnmhdr->code == ttn_needtexta && (pttta->uflags & ttf_idishwnd) || pnmhdr->code == ttn_needtextw && (ptttw->uflags & ttf_idishwnd)) { // idfrom hwnd of tool nid = ::getdlgctrlid((hwnd)nid); } //m_errprojaccel[ch] contains 1 or more error messages each...
Read more

asp.net - In javascript how to find the height and width -

-
i having .aspx , masterpage page. in masterpage loading aspx page . in aspx page design using generating table structure using code-behind server in javascript how find height , width . regards, kumar if you're looking table height , width, should it var tbl = document.getelementbyid('tbldemo'); alert(tbl.offsetwidth + '\n' + tbl.offsetheight);
Read more

c# - DataTable to EnumerableRowCollection -

-
i have datatable according client requirement can contain invalid date "00/00/1999",null,empty string.(typed datset) when collect them enumerables wish convert of invalid forms empty string. (i.e) enumerablerowcollection<datarow> query = order in _table.asenumerable() select new { orderdate= string.isnullorempty(convert.tostring(order.field<datetime>("orderdate")) || (how check date invalid date) ? string.empty : order.field<datetime>("orderdate") } how check whether date valid date or not? datetime.tryparse...like given current code: datetime date; enumerablerowcollection<datarow> query = order in _table.asenumerable() select new { orderdate= string.isnullorempty(convert.tostring(order.field<datetime>("orderdate")) || !datetime.tryparse(convert.tostring(order.field<datetime...
Read more