Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
W
webmagic
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
沈俊林
webmagic
Commits
65049bac
Commit
65049bac
authored
Jul 29, 2017
by
yihua.huang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
#571 fix spell mistake
parent
f375b9fd
Changes
4
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
21 additions
and
8 deletions
+21
-8
Page.java
webmagic-core/src/main/java/us/codecraft/webmagic/Page.java
+14
-1
Request.java
...gic-core/src/main/java/us/codecraft/webmagic/Request.java
+5
-5
HttpClientDownloader.java
...s/codecraft/webmagic/downloader/HttpClientDownloader.java
+1
-1
HttpClientDownloaderTest.java
...decraft/webmagic/downloader/HttpClientDownloaderTest.java
+1
-1
No files found.
webmagic-core/src/main/java/us/codecraft/webmagic/Page.java
View file @
65049bac
...
...
@@ -8,6 +8,7 @@ import us.codecraft.webmagic.utils.HttpConstant;
import
us.codecraft.webmagic.utils.UrlUtils
;
import
java.util.ArrayList
;
import
java.util.Arrays
;
import
java.util.List
;
import
java.util.Map
;
...
...
@@ -49,6 +50,8 @@ public class Page {
private
byte
[]
bytes
;
private
List
<
Request
>
targetRequests
=
new
ArrayList
<
Request
>();
private
String
charset
;
public
Page
()
{
}
...
...
@@ -238,6 +241,14 @@ public class Page {
this
.
bytes
=
bytes
;
}
public
String
getCharset
()
{
return
charset
;
}
public
void
setCharset
(
String
charset
)
{
this
.
charset
=
charset
;
}
@Override
public
String
toString
()
{
return
"Page{"
+
...
...
@@ -249,8 +260,10 @@ public class Page {
", url="
+
url
+
", headers="
+
headers
+
", statusCode="
+
statusCode
+
",
s
uccess="
+
downloadSuccess
+
",
downloadS
uccess="
+
downloadSuccess
+
", targetRequests="
+
targetRequests
+
", charset='"
+
charset
+
'\''
+
", bytes="
+
Arrays
.
toString
(
bytes
)
+
'}'
;
}
}
webmagic-core/src/main/java/us/codecraft/webmagic/Request.java
View file @
65049bac
...
...
@@ -49,7 +49,7 @@ public class Request implements Serializable {
* When it is set to TRUE, the downloader will not try to parse response body to text.
*
*/
private
boolean
binar
a
yContent
=
false
;
private
boolean
binaryContent
=
false
;
public
Request
()
{
}
...
...
@@ -168,12 +168,12 @@ public class Request implements Serializable {
this
.
requestBody
=
requestBody
;
}
public
boolean
isBinar
a
yContent
()
{
return
binar
a
yContent
;
public
boolean
isBinaryContent
()
{
return
binaryContent
;
}
public
void
setBinar
ayContent
(
boolean
binara
yContent
)
{
this
.
binar
ayContent
=
binara
yContent
;
public
void
setBinar
yContent
(
boolean
binar
yContent
)
{
this
.
binar
yContent
=
binar
yContent
;
}
@Override
...
...
webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java
View file @
65049bac
...
...
@@ -112,7 +112,7 @@ public class HttpClientDownloader extends AbstractDownloader {
String
contentType
=
httpResponse
.
getEntity
().
getContentType
()
==
null
?
""
:
httpResponse
.
getEntity
().
getContentType
().
getValue
();
Page
page
=
new
Page
();
page
.
setBytes
(
bytes
);
if
(!
request
.
isBinar
a
yContent
()){
if
(!
request
.
isBinaryContent
()){
page
.
setRawText
(
getResponseContent
(
charset
,
contentType
,
bytes
));
}
page
.
setUrl
(
new
PlainText
(
request
.
getUrl
()));
...
...
webmagic-core/src/test/java/us/codecraft/webmagic/downloader/HttpClientDownloaderTest.java
View file @
65049bac
...
...
@@ -280,7 +280,7 @@ public class HttpClientDownloaderTest {
public
void
run
()
throws
Exception
{
final
HttpClientDownloader
httpClientDownloader
=
new
HttpClientDownloader
();
Request
request
=
new
Request
();
request
.
setBinar
a
yContent
(
true
);
request
.
setBinaryContent
(
true
);
request
.
setUrl
(
"http://127.0.0.1:13423/"
);
Page
page
=
httpClientDownloader
.
download
(
request
,
Site
.
me
().
toTask
());
assertThat
(
page
.
getRawText
()).
isNull
();
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment