Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
W
webmagic
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
沈俊林
webmagic
Commits
de006333
Commit
de006333
authored
Jul 24, 2013
by
yihua.huang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
update java docs
parent
827972d8
Changes
5
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
5 additions
and
3 deletions
+5
-3
Spider.java
...agic-core/src/main/java/us/codecraft/webmagic/Spider.java
+1
-1
Downloader.java
...ain/java/us/codecraft/webmagic/downloader/Downloader.java
+1
-1
FileDownloader.java
...java/us/codecraft/webmagic/downloader/FileDownloader.java
+1
-0
HttpClientDownloader.java
...s/codecraft/webmagic/downloader/HttpClientDownloader.java
+1
-0
FileCacheQueueScheduler.java
...codecraft/webmagic/schedular/FileCacheQueueScheduler.java
+1
-1
No files found.
webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java
View file @
de006333
...
...
@@ -232,7 +232,7 @@ public class Spider implements Runnable, Task {
/**
* 建立多个线程下载
* @param threadNum 线程数
* @return
* @return
this
*/
public
Spider
thread
(
int
threadNum
)
{
checkIfNotRunning
();
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/Downloader.java
View file @
de006333
...
...
@@ -5,7 +5,7 @@ import us.codecraft.webmagic.Request;
import
us.codecraft.webmagic.Task
;
/**
* Downloader是webmagic下载页面的接口。webmagic默认使用了HttpComponent作为下载器,一般情况,你无需自己实现这个接口。
* Downloader是webmagic下载页面的接口。webmagic默认使用了HttpComponent作为下载器,一般情况,你无需自己实现这个接口。
<br>
* @author code4crafter@gmail.com <br>
* Date: 13-4-21
* Time: 下午12:14
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/FileDownloader.java
View file @
de006333
...
...
@@ -12,6 +12,7 @@ import us.codecraft.webmagic.selector.PlainText;
import
java.io.*
;
/**
* 使用缓存到本地的文件来模拟下载,可以在Spider框架中仅进行抽取工作。<br>
* @author code4crafer@gmail.com
* Date: 13-6-24
* Time: 上午7:24
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java
View file @
de006333
...
...
@@ -20,6 +20,7 @@ import java.io.IOException;
/**
* 封装了HttpClient的下载器。已实现指定次数重试、处理gzip、自定义UA/cookie等功能。<br>
* @author code4crafter@gmail.com <br>
* Date: 13-4-21
* Time: 下午12:15
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/schedular/FileCacheQueueScheduler.java
View file @
de006333
...
...
@@ -16,7 +16,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
import
java.util.concurrent.atomic.AtomicInteger
;
/**
* 磁盘文件实现的
安全Scheduler
,可以保证在长时间执行的任务中断后,下次启动从中断位置重新开始。<br>
* 磁盘文件实现的
url管理模块
,可以保证在长时间执行的任务中断后,下次启动从中断位置重新开始。<br>
* @author code4crafter@gmail.com <br>
* Date: 13-4-21
* Time: 下午1:13
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment