java disc based hashmap -

-

i'm working on web crawler (please don't suggest existing one, not option). have working way expected to. issue i'm using sort of server/client model server crawling , processes data, put in central location.

this location object create class wrote. internally class maintains hashmap defined hashmap<string, hashmap<string, string>>

i store data in map making url key (i keep these unique) , hasmap value stores corresponding data fields url such title,value etc

i serialize internal objects used spider multi threaded , have 5 threads crawling memory requirements go exponentially.

to far performance has been excellent hashmap, crawling 15k urls in 2.r minutes 30 seconds cpu time don't need pointed in direction of existing spider forum users have suggested.

can suggest a fast disc based solution support concurrent reading & writing? data structure doesnt have same, needs able store related meta tag values etc.

thanks in advance

i suggest using ehcache this, though you're building isn't cache. ehcache allows configure cache instance overflows disc storage, while keeping recent items in memory. can configured disc-persistent, i.e. data flushed disc on shutdown, , read memory @ startup. on top of that, it's key-value based, fits model. supports concurrent access, , since disk storage managed separate thread, shouldn't need worry disk access concurrency.

alternatively, consider proper embedded database such hypersonic (or numerous others of similar style), that's going more work.


Comments

Post a Comment

Popular posts from this blog

c++ - How do I get a multi line tooltip in MFC -

-
right now, have tool tip pops when hover on edit box. problem tool tip contains multiple error messages , in 1 long line. need have each error message on own line. error messages contained in cstring new line seperating them. my existing code below. bool ontooltiptext(uint, nmhdr* pnmhdr, lresult* presult) { assert(pnmhdr->code == ttn_needtexta || pnmhdr->code == ttn_needtextw); // need handle both ansi , unicode versions of message tooltiptexta* pttta = (tooltiptexta*)pnmhdr; tooltiptextw* ptttw = (tooltiptextw*)pnmhdr; // tchar szfulltext[256]; cstring strtiptext=_t(""); uint nid = pnmhdr->idfrom; if (pnmhdr->code == ttn_needtexta && (pttta->uflags & ttf_idishwnd) || pnmhdr->code == ttn_needtextw && (ptttw->uflags & ttf_idishwnd)) { // idfrom hwnd of tool nid = ::getdlgctrlid((hwnd)nid); } //m_errprojaccel[ch] contains 1 or more error messages each...
Read more

asp.net - In javascript how to find the height and width -

-
i having .aspx , masterpage page. in masterpage loading aspx page . in aspx page design using generating table structure using code-behind server in javascript how find height , width . regards, kumar if you're looking table height , width, should it var tbl = document.getelementbyid('tbldemo'); alert(tbl.offsetwidth + '\n' + tbl.offsetheight);
Read more

c# - DataTable to EnumerableRowCollection -

-
i have datatable according client requirement can contain invalid date "00/00/1999",null,empty string.(typed datset) when collect them enumerables wish convert of invalid forms empty string. (i.e) enumerablerowcollection<datarow> query = order in _table.asenumerable() select new { orderdate= string.isnullorempty(convert.tostring(order.field<datetime>("orderdate")) || (how check date invalid date) ? string.empty : order.field<datetime>("orderdate") } how check whether date valid date or not? datetime.tryparse...like given current code: datetime date; enumerablerowcollection<datarow> query = order in _table.asenumerable() select new { orderdate= string.isnullorempty(convert.tostring(order.field<datetime>("orderdate")) || !datetime.tryparse(convert.tostring(order.field<datetime...
Read more