JPH11328191A - Www robot retrieving system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an automatic leading-out method of a searching reference point not for simple retrieval but for retrieving only a pertinent page by estimating a WWW page updated in advance. SOLUTION: This system is provided with an update frequency arithmetic engine 22 calculating the update frequency of an optional WWW page from a retrieved result, an update expecting degree arithmetic engine 23 calculating an update expected value in an optional time from the updating frequency and a retrieving order table preparing engine 24 automatically extracting retrieving priority order. A retrieving order table is prepared by estimating whether the optional WWW page is updated at a certain time from the engine 22 and the engine 23. This WWW robot retrieving system retrieves according to this retrieving order table.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a WWW (World).
d Wide web) A method for deriving a search reference point at the time of a search by a robot, and particularly to a WWW robot detection system for obtaining an optimum search order from a plurality of search reference points serving as start points when performing a search by a WWW robot. .

[0002]

2. Description of the Related Art A WWW robot search system is called WWW.
This system has a function of detecting a structure of a WWW page inside a server and an update of each WWW page. The WWW robot search system searches a WWW server starting from a top page of a specific WWW server or a certain WWW page, and describes an HTML page from the WWW page information obtained by the search.
(HyperTextMakeup Languag
e) is analyzed to extract the position of the next WWW page hyperlinked from this WWW page. In addition,
HTML is an HTTP (Hyper) that is a protocol for communication between a client and a server in the WWW.
This is a language for describing hypertext in accordance with Text Transfer Protocol (Text Transfer Protocol).

Further, this search system stores the contents and position information of a newly created WWW page, a changed WWW page, and a deleted WWW page detected in the process of searching for a WWW server. The WWW robot performs the above-described procedure to set the W inside the target WWW server.
It manages the structure of the WW page and the update history.

However, in the conventional WWW search robot, the search start position is always specified by the operator in advance, and the search order is similarly specified by the operator in advance. The search process of the WWW search robot is performed based on a rule for sequentially analyzing HTML and acquiring a WWW page from a search start position and a search order designated by an operator in advance.

[0005] As described above, the WWW that operates according to a search rule that uses the parameters specified in advance by the operator as they are.
In robot search system, many WW
When a W page is acquired, the range and search time of the WWW robot and the load on the network are determined by the number of WWW servers to be searched and the hyperlink (HyperLi).
nk) is proportional to the sum of the depths.

In a conventional document search system, when searching for a document including a phrase specified by a user, a function of performing a full-text search on the entire stored document;
A keyword search function for searching for a document containing the instruction word by referring to an index composed of words and phrases extracted in advance from each document, and further determining which of these two functions should be used, the specified word or the like There has also been proposed a document search system provided with a judgment means for judging from conditions and performing any advantageous search according to the judgment result (for example, JP-A-10-21255). However, in this document search system, it is determined only which of the full text search and the keyword search is more advantageous, and the reference point for starting the search and the search order are executed according to a predetermined reference.

In these conventional WWW robot search systems, WWW pages that are hyperlinked sequentially are always searched from the start point specified by the operator. However, the structure of the current WWW page inside the WWW server is complicated and the hierarchy of hyperlinks is very deep, so that the operator gives a starting point in advance and sequentially searches based on this information. Has pointed out a problem that it takes a very long time to reach a WWW page actually changed inside a WWW server. Further, since the WWW server search is performed via a network, there is a problem in that unnecessary search is performed a lot to waste network resources and increase the network load.

[0008]

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems of the prior art, derive the update frequency and update expectation of each WWW page in a WWW server, and derive from these two. An object of the present invention is to provide a WWW robot search system for automatically deriving a search order of an optimum search start point and reducing a load on a network at the time of a search by a WWW robot.

[0009]

SUMMARY OF THE INVENTION The present invention automatically derives a priority determination for a WWW robot search system to search for a WWW page, and accordingly determines the actual WW.
A search for the W page is performed. More specifically, as shown in FIG. 1, the WWW robot search system includes an update frequency calculation engine 22 that calculates an update frequency of an arbitrary WWW page from a search result, and an update frequency engine at an arbitrary time based on the update frequency of the WWW page. An update expectation calculation engine 23 for calculating an update expectation value; and WWW from the update frequency and the update expectation value.
A search order table creation engine 24 for automatically extracting the search priority of the page.

[0010]

The WWW server / page structure storage unit 32 stores
WWW as a result of the search by the W server search engine 21
It stores information on WWW pages inside the server and structural information indicating connections between pages. The WWW server / page update frequency calculation engine 22 calculates the value of the update frequency of each WWW page from the structure information of the WWW page, and stores the result in the WWW server / page update frequency storage unit 33.

The WWW server / page update expectation calculation engine 23 automatically generates a parameter for calculating the degree of expectation that each WWW page is updated at a certain time from the information in the update frequency storage unit 33, and stores this parameter. W
It is stored in the WW server / page update expectation storage unit 34. The WWW server search order table creation engine 24 automatically generates a table in which WWW server pages are searched from the next time when the WWW server search engine 21 executes a search from the update frequency and the update expectation degree.

[0012]

Next, embodiments of the present invention will be described in detail with reference to the drawings. Referring to FIG.
In the first embodiment of the present invention, an input / output device 1 such as a keyboard or a display, an arithmetic processing unit 2 operated by program control, a storage device 3 for storing information, and an external WWW server via the Internet or the like are provided. And a network communication device 4 for exchanging information.

The storage device 3 includes a search schedule storage unit 3
1, a WWW server / page structure storage unit 32, a WWW server / page update frequency storage unit 33, a WWW server / page update expected value storage unit 34, and a search order table 35.

The search schedule storage unit 31 stores W
The history information of the start time at which the WW server was searched and the time required for the search are stored.

The WWW server / page structure storage unit 32
The contents of the sentences of the WWW pages arranged in the WWW server obtained by the search and the connection relations of these WWW pages are stored.

WWW server / page update frequency storage unit 33
Stores information obtained by calculating the degree of update frequency of each WWW page and quantifying it.

WWW server page update expected value storage unit 3
Numeral 4 stores information obtained by calculating the possibility that each WWW page is expected to be updated at an arbitrary time and quantifying it.

The search order table 35 stores information that is prioritized from the viewpoint of which WWW page of which WWW server to search from when performing a next search.

The calculation processing unit 2 includes a WWW server search engine 21, a WWW server / page update frequency calculation engine 22, and a WWW server / page update expected value calculation engine 2.
3. It has a WWW server search order table creation engine 24.

The WWW server search engine 21 searches for an external WWW server page via the network device 4 in accordance with the order information stored in the search order table, triggered by an execution command from the input / output device 1. I do. As a result of the search, information such as updating of WWW pages, addition / deletion of new WWW pages, and hyperlink relationships between WWW pages is stored in the WWW server / page structure storage unit 32. Information on the time at which the WWW page was searched is stored in the search schedule storage unit 31.

The WWW server page update frequency calculation engine 22 stores the WWW page update information and hyperlink information stored in the WWW server page structure storage unit 32 and the WWW server page update frequency storage unit 33. The update frequency of each WWW page is newly calculated on the basis of the information on the update frequency of each existing WWW page, and the calculation result is stored in the WWW server / page update frequency storage unit 33.

The WWW server / page update expected value calculation engine 23 includes a WWW server / page update frequency storage unit 33.
Calculates the degree that each WWW page is expected to be updated at an arbitrary time based on the information stored by the server, and stores the result as a WWW server page update expected value storage unit 34
To memorize.

The WWW server search order table creation engine 24 stores the key input from the input / output device 1 and the information stored in the WWW server / page update frequency storage unit 33 and the WWW server / page update expectation storage unit 34. From the search order table 35
To memorize.

[0024]

Next, the operation of the configuration according to the present invention will be described in detail with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG.

In response to receiving the execution instruction from the input / output device 1, the WWW server search engine 21 uses the network communication device 4 according to the search order stored in the search order table 35 in advance to send a WWW from an external WWW server.
Acquires information on the W page. WW obtained by search
The W page information includes “HTML describing the currently searched WWW page”, “whether the currently searched WWW page has been updated since the last search”, and “WWW page currently searched”. The parent WWW page that has been hyperlinked, and the child WWW page to which the currently searched WWW page is hyperlinked. The WWW server search engine 21 uses these four WWs
Information on the W page is stored in the WWW server / page structure storage unit 32.

The WWW server / page update frequency calculation engine 22 uses the information stored in the WWW server / page structure storage unit 32 to update the WWW server and the WWW page, which indicates the frequency of update of the WWW page. Is calculated and determined.

The structure of the WWW page inside the WWW server searched by the WWW server search engine 21 has the same shape as a perceptron type neural network as shown in FIG. Therefore, in the present invention, the update frequency of each WWW page is calculated by assuming that the update frequency of each WWW page (node) corresponds to the update frequency of the WWW page, assuming that the weight of each WWW page (node) corresponds to the update frequency of each WWW page. . In the case of a WWW server having a WWW page structure as shown in FIG. 2, the WWW server / page update frequency calculation engine 22 updates the update frequency of an arbitrary WWW page a with the update frequency of a child WWW page hyperlinked from the WWW page a. And the value of the update frequency of the current WWW page a using Equation (1) shown in FIG.

As a technique for deriving the update frequency of all WWW pages in the WWW server, the WWW server page update frequency calculation engine 22 focuses on an arbitrary WWW page and calculates the update frequency of the focused WWW page. Use the following method. The procedure for determining the WWW page of interest will be described below.

First, the WWW server / page update frequency calculation engine 22 stores the WWW server / page structure storage unit 3.
Paying attention to the child WWW page located at the bottom of the WWW pages stored in No. 2, the WWW server page update frequency of this page is derived. Second, the page from which the update frequency was derived is marked. Third, WWW
The server / page update frequency calculation engine 22 converts the child WWW page located at the bottom of the unmarked WWW pages among the WWW pages stored in the WWW server / page structure storage unit 32 into the next one. The target WWW page is set. Thereafter, by repeating this procedure, WW
The update frequency of the W page is derived.

If the eight WWW pages are hyperlinked in the relationship shown in FIG. 3, the calculation order is, for example, as shown in the frame 1 in FIG. 3, "page 8 → page 5 → page 6 → page 7 → Page 2 → Page 3 → Page 4 → Page 1 ”.

The WWW server / page update frequency calculation engine 22 derives the update frequency of each WWW page by the method described above, and converts the obtained update frequency to the WWW server / page.
It is stored in the page update frequency storage unit 33.

The WWW server / page update expectation calculation engine 23 calculates a value of a degree at which a WWW page can be expected to be updated at a certain point in time, and stores the result in the WWW server / page update expectation storage unit 34. Remember.

The WWW server / page update expectation engine 23 calculates the update expectation of each WWW page using the following method. When the WWW server / page update expectation calculation engine 23 calculates the update expectation of the WWW page, the progress from the time when the previous search stored in the search schedule storage unit 31 is performed to the time when the current search is performed is performed. Using the time and the update frequency w stored in the WWW server / page update frequency storage unit 33, the parameter ε of the equation (1) of FIG. 4 for deriving the update expectation Ex is converted to the equation (2) of FIG. Is calculated using The update expectation Ex at an arbitrary time t is derived from Expression (1) in FIG.

The procedure by which the WWW server search order table creation engine 24 creates search order information of WWW search robots is as follows. First, the WWW server search order table creation engine 24 copies the WWW page information stored in the WWW server / page structure storage unit 32 into the search order table 35 (Table 5-1 in FIG. 5).

Second, the WWW server search order table creation engine 24 determines the update reliability of each WWW page from the information stored in the WWW server page update expectation storage unit 34 and the search schedule storage unit 31. The calculation is performed, and the result is stored in the search order table 35 (Table 5-2 in FIG. 5).

Thirdly, the WWW server search order table creation engine 24 regards the update expectation degree of each WWW page obtained by the operation as having a higher value as a search priority and assuming that it has a higher search priority. Are sorted in descending order of priority (Table 5-3 in FIG. 5).

Fourth, the WWW server search order table creation engine 24 is operated by a human (operator) via the input / output device 1.
The WWW page information that does not meet the condition is deleted from the information stored in the search order table 35 according to the condition given by (5) (Table 5-4 in FIG. 5).

[0038]

The effects obtained by the WWW robot search system according to the present invention are as follows: the target WWW page and the child WW to which the WWW page is hyperlinked.
The purpose is to automatically generate an update expectation degree of a target WWW page at a certain time from the update state of the W page, and to determine a priority order for the next search based on the result. Thus, the WWW robot searches for the updated WWW page with a high expectation first, and thus can quickly acquire information on the updated WWW page.

In addition, when a search table is created, WWW pages that are not worth searching according to the conditions given by a human (operator) can be thinned out (Table 5-4 in FIG. 5), so unnecessary search work can be reduced and WWW can be reduced. It is possible to reduce the load on the network generated when the robot search system searches.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is an explanatory diagram illustrating a method of calculating an update frequency of an operation according to the embodiment of this invention.

FIG. 3 is an explanatory diagram showing a specific example of a calculation order at the time of calculating the update frequency of the operation according to the embodiment of the present invention;

FIG. 4 is an explanatory diagram showing a method of calculating an update expectation degree of an operation according to the embodiment of the present invention.

FIG. 5 is an explanatory diagram illustrating a search table creation method for operation according to the embodiment of this invention.

[Explanation of symbols]

 Reference Signs List 1 input / output device 2 arithmetic processing unit 3 storage device 4 network communication device 21 WWW server search engine 22 WWW server / page update frequency calculation engine 23 WWW server / page update expectation calculation engine 24 WWW server search order table creation engine 31 search schedule Storage unit 32 WWW server / page structure storage unit 33 WWW server / page update frequency storage unit 34 WWW server / page update expectation storage unit 35 WWW server search order table

Claims (6)

[Claims] 1. A WWW which performs a search according to a search order automatically generated at the time of a WWW page search.
Robot search system. 2. A WWW robot search system, wherein an optimum reference point is specified from a plurality of search reference points that can be a starting point at the time of WWW page search, and the search is performed in accordance with an automatically generated search order. 3. A WWW robot search system which automatically extracts an update frequency of an arbitrary WWW page based on the structure of the WWW page searched by the WWW robot. 4. An update frequency uniquely obtained from each WWW page based on a hyperlink relationship between WWW pages,
A WWW robot search system, which automatically extracts an update expectation at an arbitrary time. 5. A WWW robot search system, wherein a priority order of search destinations when a WWW robot searches is automatically generated from an update frequency and an expected degree of update of each WWW page. 6. An update frequency calculation engine for calculating the update frequency of each WWW page, an update expectation calculation engine for calculating an update expected value at an arbitrary time from the update frequency of the WWW page, a value of the update frequency and an update expectation WWW from degrees
A search order table creation engine for automatically extracting a search priority when a robot searches. A WWW robot search system.