Skip to content

W
webmagic
Commit 04ade756 authored by yihua.huang's avatar yihua.huang
Browse files
Options

Merge branch 'stable' of github.com:code4craft/webmagic 

Conflicts:
	README.md
	pom.xml
	webmagic-avalon/pom.xml
	webmagic-core/pom.xml
	webmagic-extension/pom.xml
	webmagic-lucene/pom.xml
	webmagic-samples/pom.xml
	webmagic-saxon/pom.xml
	webmagic-scripts/pom.xml
	webmagic-selenium/pom.xml
parents 74fdd1df 42a2676e
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 5489 additions and 1028 deletions
.gitignore
View file @ 04ade756
...@@ -2,4 +2,5 @@ target ...@@ -2,4 +2,5 @@ target
*.iml *.iml
out/ out/
.idea .idea
.classpath
.project
README.md
View file @ 04ade756
![logo](https://raw.github.com/code4craft/webmagic/master/assets/logo.jpg) ![logo](https://raw.github.com/code4craft/webmagic/master/asserts/logo.jpg)
[![Build Status](https://travis-ci.org/code4craft/webmagic.png?branch=master)](https://travis-ci.org/code4craft/webmagic) [![Build Status](https://travis-ci.org/code4craft/webmagic.png?branch=master)](https://travis-ci.org/code4craft/webmagic)
...@@ -175,6 +175,7 @@ webmagic遵循[Apache 2.0协议](http://opensource.org/licenses/Apache-2.0) ...@@ -175,6 +175,7 @@ webmagic遵循[Apache 2.0协议](http://opensource.org/licenses/Apache-2.0)
以下是为WebMagic提交过代码或者issue的朋友: 以下是为WebMagic提交过代码或者issue的朋友:
* [ccliangbo](https://github.com/ccliangbo)
* [yuany](https://github.com/yuany) * [yuany](https://github.com/yuany)
* [yxssfxwzy](https://github.com/yxssfxwzy) * [yxssfxwzy](https://github.com/yxssfxwzy)
* [linkerlin](https://github.com/linkerlin) * [linkerlin](https://github.com/linkerlin)
...@@ -188,8 +189,9 @@ webmagic遵循[Apache 2.0协议](http://opensource.org/licenses/Apache-2.0) ...@@ -188,8 +189,9 @@ webmagic遵循[Apache 2.0协议](http://opensource.org/licenses/Apache-2.0)
* [ywooer](https://github.com/ywooer) * [ywooer](https://github.com/ywooer)
* [yyw258520](https://github.com/yyw258520) * [yyw258520](https://github.com/yyw258520)
* [perfecking](https://github.com/perfecking) * [perfecking](https://github.com/perfecking)
* [ccliangbo](https://github.com/ccliangbo)
* [lidongyang](http://my.oschina.net/lidongyang) * [lidongyang](http://my.oschina.net/lidongyang)
* [seveniu](https://github.com/seveniu)
* [sebastian1118](https://github.com/sebastian1118)
### 邮件组: ### 邮件组:
...@@ -201,4 +203,4 @@ QQ: ...@@ -201,4 +203,4 @@ QQ:
### QQ群： ### QQ群：
330192938 373225642
assets/page-extract-rule.bmml 0 → 100644
View file @ 04ade756
<mockup version="1.0" skin="sketch" fontFace="Balsamiq Sans" measuredW="1154" measuredH="470" mockupW="709" mockupH="470">
<controls>
<control controlID="0" controlTypeID="com.balsamiq.mockups::BrowserWindow" x="445" y="0" w="709" h="470" measuredW="450" measuredH="400" zOrder="0" locked="false" isInGroup="-1">
<controlProperties>
<text>A%20Web%20Page%0Ahttp%3A//</text>
</controlProperties>
</control>
</controls>
</mockup>
\ No newline at end of file
en_docs/README.md
View file @ 04ade756
webmagic ![logo](https://raw.github.com/code4craft/webmagic/master/asserts/logo.jpg)
---
[Readme in Chinese](https://github.com/code4craft/webmagic/tree/master/zh_docs) [Readme in Chinese](https://github.com/code4craft/webmagic/tree/master/zh_docs)
[User Manual (Chinese)](https://github.com/code4craft/webmagic/blob/master/user-manual.md)
[![Build Status](https://travis-ci.org/code4craft/webmagic.png?branch=master)](https://travis-ci.org/code4craft/webmagic) [![Build Status](https://travis-ci.org/code4craft/webmagic.png?branch=master)](https://travis-ci.org/code4craft/webmagic)
>A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simply the development of a specific crawler. >A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.
## Features: ## Features:
...@@ -14,26 +17,19 @@ webmagic ...@@ -14,26 +17,19 @@ webmagic
* Multi-thread and Distribution support. * Multi-thread and Distribution support.
* Easy to be integrated. * Easy to be integrated.
## Install: ## Install:
Clone the repo and build: Add dependencies to your pom.xml:
git clone https://github.com/code4craft/webmagic.git
cd webmagic
mvn clean install
Add dependencies to your project:
<dependency> <dependency>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId> <artifactId>webmagic-core</artifactId>
<version>0.4.2</version> <version>0.4.3</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId> <artifactId>webmagic-extension</artifactId>
<version>0.4.2</version> <version>0.4.3</version>
</dependency> </dependency>
## Get Started: ## Get Started:
...@@ -42,10 +38,10 @@ Add dependencies to your project: ...@@ -42,10 +38,10 @@ Add dependencies to your project:
Write a class implements PageProcessor： Write a class implements PageProcessor：
```java
public class OschinaBlogPageProcesser implements PageProcessor { public class OschinaBlogPageProcesser implements PageProcessor {
private Site site = Site.me().setDomain("my.oschina.net") private Site site = Site.me().setDomain("my.oschina.net");
.addStartUrl("http://my.oschina.net/flashsword/blog");
@Override @Override
public void process(Page page) { public void process(Page page) {
...@@ -63,10 +59,11 @@ Write a class implements PageProcessor： ...@@ -63,10 +59,11 @@ Write a class implements PageProcessor：
} }
public static void main(String[] args) { public static void main(String[] args) {
Spider.create(new OschinaBlogPageProcesser()) Spider.create(new OschinaBlogPageProcesser()).addUrl("http://my.oschina.net/flashsword/blog")
.pipeline(new ConsolePipeline()).run(); .addPipeline(new ConsolePipeline()).run();
} }
} }
```
* `page.addTargetRequests(links)` * `page.addTargetRequests(links)`
...@@ -74,6 +71,7 @@ Write a class implements PageProcessor： ...@@ -74,6 +71,7 @@ Write a class implements PageProcessor：
You can also use annotation way: You can also use annotation way:
```java
@TargetUrl("http://my.oschina.net/flashsword/blog/\\d+") @TargetUrl("http://my.oschina.net/flashsword/blog/\\d+")
public class OschinaBlog { public class OschinaBlog {
...@@ -88,10 +86,11 @@ You can also use annotation way: ...@@ -88,10 +86,11 @@ You can also use annotation way:
public static void main(String[] args) { public static void main(String[] args) {
OOSpider.create( OOSpider.create(
Site.me().addStartUrl("http://my.oschina.net/flashsword/blog"), Site.me(),
new ConsolePageModelPipeline(), OschinaBlog.class).run(); new ConsolePageModelPipeline(), OschinaBlog.class).addUrl("http://my.oschina.net/flashsword/blog").run();
} }
} }
```
### Docs and samples: ### Docs and samples:
...@@ -103,11 +102,30 @@ Javadocs: [http://code4craft.github.io/webmagic/docs/en/](http://code4craft.gith ...@@ -103,11 +102,30 @@ Javadocs: [http://code4craft.github.io/webmagic/docs/en/](http://code4craft.gith
There are some samples in `webmagic-samples` package. There are some samples in `webmagic-samples` package.
### Lisence: ### Lisence:
Lisenced under [Apache 2.0 lisence](http://opensource.org/licenses/Apache-2.0) Lisenced under [Apache 2.0 lisence](http://opensource.org/licenses/Apache-2.0)
### Contributors:
Thanks these people for commiting source code, reporting bugs or suggesting for new feature:
* [yuany](https://github.com/yuany)
* [yxssfxwzy](https://github.com/yxssfxwzy)
* [linkerlin](https://github.com/linkerlin)
* [d0ngw](https://github.com/d0ngw)
* [xuchaoo](https://github.com/xuchaoo)
* [supermicah](https://github.com/supermicah)
* [SimpleExpress](https://github.com/SimpleExpress)
* [aruanruan](https://github.com/aruanruan)
* [l1z2g9](https://github.com/l1z2g9)
* [zhegexiaohuozi](https://github.com/zhegexiaohuozi)
* [ywooer](https://github.com/ywooer)
* [yyw258520](https://github.com/yyw258520)
* [perfecking](https://github.com/perfecking)
* [lidongyang](http://my.oschina.net/lidongyang)
### Thanks: ### Thanks:
To write webmagic, I refered to the projects below : To write webmagic, I refered to the projects below :
...@@ -124,3 +142,10 @@ To write webmagic, I refered to the projects below : ...@@ -124,3 +142,10 @@ To write webmagic, I refered to the projects below :
[https://gitcafe.com/laiweiwei/Spiderman](https://gitcafe.com/laiweiwei/Spiderman) [https://gitcafe.com/laiweiwei/Spiderman](https://gitcafe.com/laiweiwei/Spiderman)
### Mail-list:
[https://groups.google.com/forum/#!forum/webmagic-java](https://groups.google.com/forum/#!forum/webmagic-java)
[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/code4craft/webmagic/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
pom.xml
View file @ 04ade756
...@@ -6,7 +6,7 @@ ...@@ -6,7 +6,7 @@
<version>7</version> <version>7</version>
</parent> </parent>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<version>0.4.3</version> <version>0.5.0</version>
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging> <packaging>pom</packaging>
<properties> <properties>
...@@ -51,11 +51,10 @@ ...@@ -51,11 +51,10 @@
<module>webmagic-core</module> <module>webmagic-core</module>
<module>webmagic-extension/</module> <module>webmagic-extension/</module>
<module>webmagic-scripts/</module> <module>webmagic-scripts/</module>
<module>webmagic-avalon</module>
<module>webmagic-lucene</module>
<module>webmagic-samples</module>
<module>webmagic-saxon</module>
<module>webmagic-selenium</module> <module>webmagic-selenium</module>
<module>webmagic-saxon</module>
<module>webmagic-samples</module>
<module>webmagic-avalon</module>
</modules> </modules>
<dependencyManagement> <dependencyManagement>
...@@ -63,7 +62,7 @@ ...@@ -63,7 +62,7 @@
<dependency> <dependency>
<groupId>junit</groupId> <groupId>junit</groupId>
<artifactId>junit</artifactId> <artifactId>junit</artifactId>
<version>4.7</version> <version>4.11</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency> <dependency>
...@@ -89,12 +88,7 @@ ...@@ -89,12 +88,7 @@
<dependency> <dependency>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>xsoup</artifactId> <artifactId>xsoup</artifactId>
<version>0.2.0</version> <version>0.2.2</version>
</dependency>
<dependency>
<groupId>net.sf.saxon</groupId>
<artifactId>Saxon-HE</artifactId>
<version>9.5.1-1</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>com.alibaba</groupId> <groupId>com.alibaba</groupId>
...@@ -121,11 +115,6 @@ ...@@ -121,11 +115,6 @@
<artifactId>commons-collections</artifactId> <artifactId>commons-collections</artifactId>
<version>3.2.1</version> <version>3.2.1</version>
</dependency> </dependency>
<dependency>
<groupId>net.sourceforge.htmlcleaner</groupId>
<artifactId>htmlcleaner</artifactId>
<version>2.5</version>
</dependency>
<dependency> <dependency>
<groupId>org.apache.commons</groupId> <groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId> <artifactId>commons-io</artifactId>
...@@ -136,6 +125,12 @@ ...@@ -136,6 +125,12 @@
<artifactId>jsoup</artifactId> <artifactId>jsoup</artifactId>
<version>1.7.2</version> <version>1.7.2</version>
</dependency> </dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.5</version>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
</dependencyManagement> </dependencyManagement>
...@@ -159,26 +154,26 @@ ...@@ -159,26 +154,26 @@
<encoding>UTF-8</encoding> <encoding>UTF-8</encoding>
</configuration> </configuration>
</plugin> </plugin>
<plugin> <!--<plugin>-->
<groupId>org.apache.maven.plugins</groupId> <!--<groupId>org.apache.maven.plugins</groupId>-->
<artifactId>maven-dependency-plugin</artifactId> <!--<artifactId>maven-dependency-plugin</artifactId>-->
<version>2.8</version> <!--<version>2.8</version>-->
<executions> <!--<executions>-->
<execution> <!--<execution>-->
<id>copy-dependencies</id> <!--<id>copy-dependencies</id>-->
<phase>package</phase> <!--<phase>package</phase>-->
<goals> <!--<goals>-->
<goal>copy-dependencies</goal> <!--<goal>copy-dependencies</goal>-->
</goals> <!--</goals>-->
<configuration> <!--<configuration>-->
<outputDirectory>${project.build.directory}/lib</outputDirectory> <!--<outputDirectory>${project.build.directory}/lib</outputDirectory>-->
<overWriteReleases>false</overWriteReleases> <!--<overWriteReleases>false</overWriteReleases>-->
<overWriteSnapshots>false</overWriteSnapshots> <!--<overWriteSnapshots>false</overWriteSnapshots>-->
<overWriteIfNewer>true</overWriteIfNewer> <!--<overWriteIfNewer>true</overWriteIfNewer>-->
</configuration> <!--</configuration>-->
</execution> <!--</execution>-->
</executions> <!--</executions>-->
</plugin> <!--</plugin>-->
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId> <artifactId>maven-resources-plugin</artifactId>
...@@ -187,6 +182,15 @@ ...@@ -187,6 +182,15 @@
<encoding>UTF-8</encoding> <encoding>UTF-8</encoding>
</configuration> </configuration>
</plugin> </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<excludes>
<exclude>log4j.xml</exclude>
</excludes>
</configuration>
</plugin>
<plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId> <artifactId>maven-source-plugin</artifactId>
......
user-manual.md
View file @ 04ade756
...@@ -65,7 +65,7 @@ webmagic还包含两个可用的扩展包，因为这两个包都依赖了比较 ...@@ -65,7 +65,7 @@ webmagic还包含两个可用的扩展包，因为这两个包都依赖了比较
git clone http://git.oschina.net/flashsword20/webmagic.git git clone http://git.oschina.net/flashsword20/webmagic.git
在**bin/lib**目录下，有项目依赖的所有jar包，直接在IDE里import即可。 在**lib**目录下，有项目依赖的所有jar包，直接在IDE里import即可。
-------- --------
......
webmagic-avalon/README.md 0 → 100644
View file @ 04ade756
WebMagic-Avalon
========
> Spiders Manage Web
see [#issue43](https://github.com/code4craft/webmagic/issues/43)
\ No newline at end of file
webmagic-avalon/forger/LICENSE 0 → 100644
View file @ 04ade756
This diff is collapsed.
webmagic-avalon/forger/README.md 0 → 100644
View file @ 04ade756
forger
======
Dynamic Java object generator with template class and configuration.
## Compiler
Use groovy compiler. Compile source code to Java class.
## PropertyLoader
Load properties of object from user input.
## API
```java
@Test
public void testForgerCreateByClassAnnotationCompile() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), new GroovyForgerCompiler());
Forger<Fooable> forger = forgerFactory.<Fooable>compile(Foo.SOURCE_CODE);
Fooable foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
Field field = forger.getClazz().getDeclaredField("foo");
field.setAccessible(true);
assertThat(field.get(foo)).isEqualTo("test");
assertThat(foo.foo()).isEqualTo("test");
}
```
\ No newline at end of file
webmagic-avalon/forger/pom.xml 0 → 100644
View file @ 04ade756
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>org.sonatype.oss</groupId>
<artifactId>oss-parent</artifactId>
<version>7</version>
</parent>
<groupId>us.codecraft</groupId>
<artifactId>forger</artifactId>
<version>0.1.1-SNAPSHOT</version>
<modelVersion>4.0.0</modelVersion>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<name>forger</name>
<description>
Dynamic Java object generator with template class and configuration.
</description>
<url>https://github.com/code4craft/forger/</url>
<developers>
<developer>
<id>code4craft</id>
<name>Yihua huang</name>
<email>code4crafer@gmail.com</email>
</developer>
</developers>
<scm>
<connection>scm:git:git@github.com:code4craft/forger.git</connection>
<developerConnection>scm:git:git@github.com:code4craft/forger.git</developerConnection>
<url>git@github.com:code4craft/forger.git</url>
<tag>HEAD</tag>
</scm>
<licenses>
<license>
<name>Apache License,Version 2</name>
<url>http://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>15.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>2.6</version>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.4.1</version>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>release-sign-artifacts</id>
<activation>
<property>
<name>performRelease</name>
<value>true</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.1</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
webmagic-avalon/forger/src/main/java/us/codecraft/forger/Forger.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
import us.codecraft.forger.property.Property;
import us.codecraft.forger.property.PropertyLoader;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public class Forger<T> {
private final Class<T> clazz;
private final PropertyLoader propertyLoader;
public Forger(Class<T> clazz,PropertyLoader propertyLoader) {
this.clazz = clazz;
this.propertyLoader = propertyLoader;
}
public T forge(Map<String, Object> properties) throws IllegalAccessException, InstantiationException {
T t = clazz.newInstance();
propertyLoader.load(t, properties);
return t;
}
public List<Property> getPropertyNames() {
return propertyLoader.getProperties(clazz);
}
public Class<T> getClazz() {
return clazz;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/ForgerFactory.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
import us.codecraft.forger.compiler.ForgerCompiler;
import us.codecraft.forger.property.PropertyLoader;
/**
* @author code4crafter@gmail.com
*/
public class ForgerFactory {
private final PropertyLoader propertyLoader;
private final ForgerCompiler forgerCompiler;
public ForgerFactory(PropertyLoader propertyLoader, ForgerCompiler forgerCompiler) {
this.propertyLoader = propertyLoader;
this.forgerCompiler = forgerCompiler;
}
public <T> Forger<T> compile(String sourceCode) {
Class clazz = forgerCompiler.compile(sourceCode);
return new Forger(clazz, propertyLoader);
}
public <T> Forger<T> create(Class clazz) {
return new Forger(clazz, propertyLoader);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/compiler/ForgerCompiler.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.compiler;
/**
* @author code4crafter@gmail.com
*/
public interface ForgerCompiler {
public Class compile(String sourceCode);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/compiler/GroovyForgerCompiler.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.compiler;
import groovy.lang.GroovyClassLoader;
/**
* @author code4crafter@gmail.com
*/
public class GroovyForgerCompiler implements ForgerCompiler{
private GroovyClassLoader groovyClassLoader = new GroovyClassLoader();
@Override
public Class compile(String sourceCode) {
return groovyClassLoader.parseClass(sourceCode);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/AbstractPropertyLoader.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import us.codecraft.forger.property.format.*;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public abstract class AbstractPropertyLoader implements PropertyLoader {
private TypeFormatterFactory typeFormatterFactory = new TypeFormatterFactory();
protected Logger logger = LoggerFactory.getLogger(getClass());
protected TypeFormatterFactory getTypeFormatterFactory() {
return typeFormatterFactory;
}
@Override
public <T> T load(T object, Map<String, Object> propertyConfigs) {
List<Property> properties = getProperties(object.getClass());
for (Property property : properties) {
Object value = propertyConfigs.get(property.getName());
if (value == null) {
throw new IllegalArgumentException("Config for property " + property.getName() + " is missing!");
}
ObjectFormatter objectFormatter = property.getObjectFormatter();
switch (property.getType()) {
case PropertyString:
Object fieldValue = objectFormatter.format(String.valueOf(value));
try {
property.getField().set(object, fieldValue);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
case PropertyList:
if (!List.class.isAssignableFrom(value.getClass())) {
throw new IllegalArgumentException("Config for property " + property.getName() + " should be subclass of List!");
}
List listField = new ArrayList();
List<String> listConfigs = (List) value;
for (String listConfig : listConfigs) {
listField.add(objectFormatter.format(listConfig));
}
try {
property.getField().set(object, listField);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
case PropertyMap:
if (!Map.class.isAssignableFrom(value.getClass())) {
throw new IllegalArgumentException("Config for property " + property.getName() + " should be subclass of List!");
}
Map mapField = new HashMap();
Map<String, String> mapConfigs = (Map<String, String>) value;
for (Map.Entry<String, String> entry : mapConfigs.entrySet()) {
mapField.put(entry.getKey(), objectFormatter.format(entry.getValue()));
}
try {
property.getField().set(object, mapField);
} catch (IllegalAccessException e) {
logger.warn("Set field " + property.getField() + " error!", e);
}
break;
}
}
return object;
}
protected ObjectFormatter prepareTypeFormatterParam(TypeFormatter objectFormatter, String[] params) {
if (params == null) {
return objectFormatter;
}
return new ObjectFormatterWithParams().setTypeFormatter(objectFormatter).setParams(params);
}
protected ObjectFormatter getObjectFormatter(Field field) {
Class type = field.getType();
if (List.class.isAssignableFrom(type) || Map.class.isAssignableFrom(type)) {
type = String.class;
}
if (field.isAnnotationPresent(Formatter.class)) {
Formatter formatter = field.getAnnotation(Formatter.class);
if (!formatter.formatter().equals(TypeFormatter.class)) {
TypeFormatter typeFormatter = typeFormatterFactory.getByFormatterClass(formatter.formatter());
if (typeFormatter != null) {
return prepareTypeFormatterParam(typeFormatter,formatter.value());
}
typeFormatterFactory.put(formatter.formatter());
return prepareTypeFormatterParam(typeFormatterFactory.getByFormatterClass(formatter.formatter()), formatter.value());
} else if (!formatter.subClazz().equals(String.class)) {
type = formatter.subClazz();
TypeFormatter typeFormatter = typeFormatterFactory.get(type);
if (typeFormatter == null) {
throw new IllegalArgumentException("No typeFormatter for class " + type);
}
return prepareTypeFormatterParam(typeFormatter, formatter.value());
}
}
return getTypeFormatterFactory().get(BasicTypeFormatter.detectBasicClass(type));
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/AnnotationPropertyLoader.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;
/**
* @author code4crafter@gmail.com
*/
public class AnnotationPropertyLoader extends AbstractPropertyLoader {
@Override
public List<Property> getProperties(Class clazz) {
Field[] fields = clazz.getDeclaredFields();
List<Property> properties = new ArrayList<Property>(fields.length);
for (Field field : fields) {
Inject inject = field.getAnnotation(Inject.class);
if (inject != null) {
if (!field.isAccessible()) {
field.setAccessible(true);
}
Property property = Property.fromField(field);
if (inject.value().length() > 0) {
property.setName(inject.value());
}
property.setObjectFormatter(getObjectFormatter(field));
properties.add(property);
}
}
return properties;
}
}
webmagic-extension/src/main/java/us/codecraft/webmagic/configurable/Inject.java → webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/Inject.java
View file @ 04ade756
package us.codecraft.webmagic.configurable; package us.codecraft.forger.property;
import java.lang.annotation.ElementType; import java.lang.annotation.ElementType;
import java.lang.annotation.Retention; import java.lang.annotation.Retention;
import java.lang.annotation.Target; import java.lang.annotation.Target;
/** /**
* @author yihua.huang@dianping.com * @author code4crafter@gmail.com
*/ */
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME) @Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD}) @Target({ElementType.FIELD})
public @interface Inject { public @interface Inject {
String value() default ""; String value() default "";
} }
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/Property.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import us.codecraft.forger.property.format.ObjectFormatter;
import java.lang.reflect.Field;
/**
* @author code4crafter@gmail.com
*/
public class Property {
private String name;
private PropertyType type;
private Field field;
private ObjectFormatter objectFormatter;
public ObjectFormatter getObjectFormatter() {
return objectFormatter;
}
public Property setObjectFormatter(ObjectFormatter objectFormatter) {
this.objectFormatter = objectFormatter;
return this;
}
public String getName() {
return name;
}
public Property setName(String name) {
this.name = name;
return this;
}
public PropertyType getType() {
return type;
}
public Property setType(PropertyType type) {
this.type = type;
return this;
}
public Field getField() {
return field;
}
public Property setField(Field field) {
this.field = field;
return this;
}
public static Property fromField(Field field) {
return new Property().setName(field.getName()).setType(PropertyType.from(field.getType())).setField(field);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/PropertyLoader.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public interface PropertyLoader {
public <T> T load(T object, Map<String, Object> propertyConfigs);
public List<Property> getProperties(Class clazz);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/PropertyType.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public enum PropertyType {
PropertyString,PropertyMap,PropertyList;
public static PropertyType from(Class clazz){
if (Map.class.isAssignableFrom(clazz)){
return PropertyMap;
}
if (List.class.isAssignableFrom(clazz)){
return PropertyList;
}
return PropertyString;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/SimpleFieldPropertyLoader.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.List;
/**
* @author code4crafter@gmail.com
*/
public class SimpleFieldPropertyLoader extends AbstractPropertyLoader {
@Override
public List<Property> getProperties(Class clazz) {
Field[] fields = clazz.getDeclaredFields();
List<Property> properties = new ArrayList<Property>(fields.length);
for (Field field : fields) {
if (Modifier.isStatic(field.getModifiers())){
continue;
}
if (!field.isAccessible()){
field.setAccessible(true);
}
properties.add(Property.fromField(field).setObjectFormatter(getObjectFormatter(field)));
}
return properties;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/BasicTypeFormatter.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
import java.util.Arrays;
import java.util.List;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public abstract class BasicTypeFormatter<T> implements TypeFormatter<T> {
@Override
public T format(String text) {
if (text == null) {
return null;
}
text = text.trim();
return formatTrimmed(text);
}
@Override
public T format(String text, String[] params) {
return format(text);
}
protected abstract T formatTrimmed(String raw);
public static final List<Class<? extends TypeFormatter>> basicTypeFormatters = Arrays.<Class<? extends TypeFormatter>>asList(IntegerFormatter.class,
LongFormatter.class, DoubleFormatter.class, FloatFormatter.class, ShortFormatter.class,
CharactorFormatter.class, ByteFormatter.class, BooleanFormatter.class, DateFormatter.class, StringFormatter.class);
public static Class<?> detectBasicClass(Class<?> type) {
if (type.equals(Integer.TYPE) || type.equals(Integer.class)) {
return Integer.class;
} else if (type.equals(Long.TYPE) || type.equals(Long.class)) {
return Long.class;
} else if (type.equals(Double.TYPE) || type.equals(Double.class)) {
return Double.class;
} else if (type.equals(Float.TYPE) || type.equals(Float.class)) {
return Float.class;
} else if (type.equals(Short.TYPE) || type.equals(Short.class)) {
return Short.class;
} else if (type.equals(Character.TYPE) || type.equals(Character.class)) {
return Character.class;
} else if (type.equals(Byte.TYPE) || type.equals(Byte.class)) {
return Byte.class;
} else if (type.equals(Boolean.TYPE) || type.equals(Boolean.class)) {
return Boolean.class;
}
return type;
}
public static class IntegerFormatter extends BasicTypeFormatter<Integer> {
@Override
public Integer formatTrimmed(String raw) {
return Integer.parseInt(raw);
}
@Override
public Class<Integer> clazz() {
return Integer.class;
}
}
public static class LongFormatter extends BasicTypeFormatter<Long> {
@Override
public Long formatTrimmed(String raw) {
return Long.parseLong(raw);
}
@Override
public Class<Long> clazz() {
return Long.class;
}
}
public static class DoubleFormatter extends BasicTypeFormatter<Double> {
@Override
public Double formatTrimmed(String raw) {
return Double.parseDouble(raw);
}
@Override
public Class<Double> clazz() {
return Double.class;
}
}
public static class FloatFormatter extends BasicTypeFormatter<Float> {
@Override
public Float formatTrimmed(String raw) {
return Float.parseFloat(raw);
}
@Override
public Class<Float> clazz() {
return Float.class;
}
}
public static class ShortFormatter extends BasicTypeFormatter<Short> {
@Override
public Short formatTrimmed(String raw) {
return Short.parseShort(raw);
}
@Override
public Class<Short> clazz() {
return Short.class;
}
}
public static class CharactorFormatter extends BasicTypeFormatter<Character> {
@Override
public Character formatTrimmed(String raw) {
return raw.charAt(0);
}
@Override
public Class<Character> clazz() {
return Character.class;
}
}
public static class ByteFormatter extends BasicTypeFormatter<Byte> {
@Override
public Byte formatTrimmed(String raw) {
return Byte.parseByte(raw, 10);
}
@Override
public Class<Byte> clazz() {
return Byte.class;
}
}
public static class BooleanFormatter extends BasicTypeFormatter<Boolean> {
@Override
public Boolean formatTrimmed(String raw) {
return Boolean.parseBoolean(raw);
}
@Override
public Class<Boolean> clazz() {
return Boolean.class;
}
}
public static class StringFormatter implements TypeFormatter<String> {
@Override
public String format(String text) {
return text;
}
@Override
public String format(String text, String[] params) {
return format(text);
}
@Override
public Class<String> clazz() {
return String.class;
}
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/DateFormatter.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
import org.apache.commons.lang3.time.DateUtils;
import java.text.ParseException;
import java.util.Date;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public class DateFormatter implements TypeFormatter<Date> {
public static final String[] DEFAULT_PATTERN = new String[]{"yyyy-MM-dd HH:mm"};
@Override
public Date format(String text) {
return format(text,DEFAULT_PATTERN);
}
@Override
public Date format(String text, String[] params) {
try {
return DateUtils.parseDate(text, params);
} catch (ParseException e) {
throw new IllegalArgumentException(e);
}
}
@Override
public Class<Date> clazz() {
return Date.class;
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/Formatter.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
/**
* Define how the result string is convert to an object for field.
*
* @author code4crafter@gmail.com <br>
* @since 0.3.2
*/
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD})
public @interface Formatter {
/**
* Set formatter params.
*
* @return formatter params
*/
String[] value();
/**
* Specific the class of field of class of elements in collection for field. <br/>
* It is not necessary to be set because we can detect the class by class of field,
* unless you use a collection as a field. <br/>
*
* @return the class of field
*/
Class subClazz() default String.class;
/**
* If there are more than one formatter for a class, just specify the implement.
* @return implement
*/
Class<? extends TypeFormatter> formatter() default TypeFormatter.class;
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/ObjectFormatter.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public interface ObjectFormatter<T> {
T format(String text);
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/ObjectFormatterWithParams.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public class ObjectFormatterWithParams<T> implements ObjectFormatter<T> {
private TypeFormatter<T> typeFormatter;
private String[] params;
public TypeFormatter getTypeFormatter() {
return typeFormatter;
}
public ObjectFormatterWithParams<T> setTypeFormatter(TypeFormatter typeFormatter) {
this.typeFormatter = typeFormatter;
return this;
}
public String[] getParams() {
return params;
}
public ObjectFormatterWithParams setParams(String[] params) {
this.params = params;
return this;
}
@Override
public T format(String text) {
return typeFormatter.format(text, params);
}
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/TypeFormatter.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
/**
* @author code4crafter@gmail.com
*/
public interface TypeFormatter<T> extends ObjectFormatter<T> {
T format(String text, String[] params);
Class<T> clazz();
}
webmagic-avalon/forger/src/main/java/us/codecraft/forger/property/format/TypeFormatterFactory.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.property.format;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* @author code4crafter@gmail.com
* @since 0.3.2
*/
public class TypeFormatterFactory {
private Logger logger = LoggerFactory.getLogger(getClass());
private Map<Class, TypeFormatter> objectFormatterMapWithPropertyAsKey = new ConcurrentHashMap<Class, TypeFormatter>();
private Map<Class, TypeFormatter> objectFormatterMapWithClassAsKey = new ConcurrentHashMap<Class, TypeFormatter>();
public TypeFormatterFactory() {
initFormatterMap();
}
private void initFormatterMap() {
for (Class<? extends TypeFormatter> basicTypeFormatter : BasicTypeFormatter.basicTypeFormatters) {
put(basicTypeFormatter);
}
put(DateFormatter.class);
}
public synchronized void put(Class<? extends TypeFormatter> objectFormatterClazz) {
try {
TypeFormatter typeFormatter = objectFormatterClazz.newInstance();
if (typeFormatter.clazz() != null) {
objectFormatterMapWithPropertyAsKey.put(typeFormatter.clazz(), typeFormatter);
}
objectFormatterMapWithClassAsKey.put(objectFormatterClazz, typeFormatter);
} catch (InstantiationException e) {
logger.error("Init objectFormatter error", e);
} catch (IllegalAccessException e) {
logger.error("Init objectFormatter error", e);
}
}
public TypeFormatter get(Class<?> clazz) {
return objectFormatterMapWithPropertyAsKey.get(clazz);
}
public TypeFormatter getByFormatterClass(Class<?> clazz) {
return objectFormatterMapWithClassAsKey.get(clazz);
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Bar.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
import us.codecraft.forger.property.Inject;
import us.codecraft.forger.property.format.Formatter;
import java.util.List;
import java.util.Map;
/**
* @author code4crafter@gmail.com
*/
public class Bar {
@Inject("bar")
private String bar;
@Inject
private List<String> values;
@Formatter(value = "", subClazz = Integer.class)
@Inject
private Map<String, Integer> idMap;
public String getBar() {
return bar;
}
public void setBar(String bar) {
this.bar = bar;
}
public List<String> getValues() {
return values;
}
public void setValues(List<String> values) {
this.values = values;
}
public Map<String, Integer> getIdMap() {
return idMap;
}
public void setIdMap(Map<String, Integer> idMap) {
this.idMap = idMap;
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Foo.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
import us.codecraft.forger.property.Inject;
import us.codecraft.forger.property.format.Formatter;
/**
* @author code4crafter@gmail.com
*/
public class Foo implements Fooable{
@Formatter("")
@Inject("fooa")
private String foo;
public static final String SOURCE_CODE="import us.codecraft.forger.*;\n" +
"import us.codecraft.forger.property.Inject;\n" +
"import us.codecraft.forger.property.Inject;\n" +
"import us.codecraft.forger.property.format.Formatter;\n" +
"\n" +
"/**\n" +
" * @author code4crafter@gmail.com\n" +
" */\n" +
"public class Foo implements Fooable{\n" +
"\n" +
" @Formatter(\"\")\n" +
" @Inject(\"fooa\")\n" +
" private String foo;\n" +
"\n" +
" public String getFoo() {\n" +
" return foo;\n" +
" }\n" +
"\n" +
" @Override\n" +
" public String foo() {\n" +
" return foo;\n" +
" }\n" +
"}";
public String getFoo() {
return foo;
}
@Override
public String foo() {
return foo;
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/Fooable.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
/**
* @author code4crafter@gmail.com
*/
public interface Fooable {
public String foo();
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/ForgerFactoryTest.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger;
import com.google.common.collect.ImmutableMap;
import org.junit.Test;
import us.codecraft.forger.compiler.GroovyForgerCompiler;
import us.codecraft.forger.property.AnnotationPropertyLoader;
import us.codecraft.forger.property.SimpleFieldPropertyLoader;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.*;
/**
* @author code4crafter@gmail.com
*/
public class ForgerFactoryTest {
@Test
public void testForgerCreateByClassProperty() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new SimpleFieldPropertyLoader(), null);
Forger<Foo> forger = forgerFactory.<Foo>create(Foo.class);
Foo foo = forger.forge(ImmutableMap.<String, Object>of("foo", "test"));
assertThat(foo.getFoo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotation() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), null);
Forger<Foo> forger = forgerFactory.<Foo>create(Foo.class);
Foo foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
assertThat(foo.getFoo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotationCompile() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), new GroovyForgerCompiler());
Forger<Fooable> forger = forgerFactory.<Fooable>compile(Foo.SOURCE_CODE);
Fooable foo = forger.forge(ImmutableMap.<String, Object>of("fooa", "test"));
Field field = forger.getClazz().getDeclaredField("foo");
field.setAccessible(true);
assertThat(field.get(foo)).isEqualTo("test");
assertThat(foo.foo()).isEqualTo("test");
}
@Test
public void testForgerCreateByClassAnnotationWithCollections() throws Exception {
ForgerFactory forgerFactory = new ForgerFactory(new AnnotationPropertyLoader(), null);
Forger<Bar> forger = forgerFactory.<Bar>create(Bar.class);
Map<String, Object> map = new HashMap<String, Object>();
map.put("bar", "bar");
Map<String, String> submap = new HashMap<String, String>();
submap.put("1", "1");
submap.put("2", "2");
map.put("idMap", submap);
List<String> sublist = new ArrayList<String>();
sublist.add("test");
map.put("values", sublist);
Bar forge = forger.forge(map);
assertThat(forge.getValues().size() > 0);
assertThat(forge.getIdMap().get("1")).isEqualTo(1);
}
}
webmagic-avalon/forger/src/test/java/us/codecraft/forger/compiler/GroovyForgerCompilerTest.java 0 → 100644
View file @ 04ade756
package us.codecraft.forger.compiler;
import org.junit.Test;
import us.codecraft.forger.Foo;
import static org.assertj.core.api.Assertions.assertThat;
/**
* @author code4crafter@gmail.com
*/
public class GroovyForgerCompilerTest {
@Test
public void testGroovyClassLoader() throws Exception {
GroovyForgerCompiler groovyForgerCompiler = new GroovyForgerCompiler();
Class compiledClass = groovyForgerCompiler.compile(Foo.SOURCE_CODE);
assertThat(compiledClass.getName()).isEqualTo("Foo");
}
}
webmagic-avalon/forger/src/test/resources/log4j.xml 0 → 100644
View file @ 04ade756
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yy-MM-dd HH:mm:ss,SSS} %-5p %c(%F:%L) ## %m%n" />
</layout>
</appender>
<logger name="org.springframework" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<logger name="org.apache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<logger name="net.sf.ehcache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<root>
<level value="info" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>
webmagic-avalon/pom.xml
View file @ 04ade756
...@@ -3,94 +3,130 @@ ...@@ -3,94 +3,130 @@
<parent> <parent>
<artifactId>webmagic-parent</artifactId> <artifactId>webmagic-parent</artifactId>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<version>0.4.3</version> <version>0.5.0</version>
</parent> </parent>
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>webmagic-avalon</artifactId> <artifactId>webmagic-avalon</artifactId>
<packaging>war</packaging> <packaging>pom</packaging>
<dependencies> <modules>
<dependency> <module>forger</module>
<groupId>us.codecraft</groupId> <module>webmagic-admin</module>
<artifactId>webmagic-scripts</artifactId> <module>webmagic-worker</module>
<version>${project.version}</version> <module>webmagic-avalon-common</module>
</dependency> </modules>
<dependency> <dependencyManagement>
<groupId>org.mybatis</groupId> <dependencies>
<artifactId>mybatis</artifactId>
<version>3.1.1</version> <dependency>
</dependency> <groupId>us.codecraft</groupId>
<artifactId>webmagic-scripts</artifactId>
<dependency> <version>${project.version}</version>
<groupId>org.mybatis</groupId> </dependency>
<artifactId>mybatis-spring</artifactId>
<version>1.1.1</version> <dependency>
</dependency> <groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<dependency> <version>3.1.1</version>
<groupId>org.freemarker</groupId> </dependency>
<artifactId>freemarker</artifactId>
<version>2.3.19</version> <dependency>
</dependency> <groupId>org.mybatis</groupId>
<dependency> <artifactId>mybatis-spring</artifactId>
<groupId>org.springframework</groupId> <version>1.1.1</version>
<artifactId>spring-test</artifactId> </dependency>
<version>${spring-version}</version>
<scope>test</scope> <dependency>
</dependency> <groupId>us.codecraft</groupId>
<artifactId>forger</artifactId>
<dependency> <version>0.1.1-SNAPSHOT</version>
<groupId>org.springframework</groupId> </dependency>
<artifactId>spring-aop</artifactId>
<version>${spring-version}</version> <dependency>
</dependency> <groupId>org.freemarker</groupId>
<artifactId>freemarker</artifactId>
<dependency> <version>2.3.19</version>
<groupId>org.aspectj</groupId> </dependency>
<artifactId>aspectjrt</artifactId>
<version>1.7.2</version> <dependency>
</dependency> <groupId>org.springframework</groupId>
<dependency> <artifactId>spring-test</artifactId>
<groupId>org.aspectj</groupId> <version>${spring-version}</version>
<artifactId>aspectjweaver</artifactId> <scope>test</scope>
<version>1.7.2</version> </dependency>
</dependency>
<dependency> <dependency>
<groupId>org.springframework</groupId> <groupId>org.assertj</groupId>
<artifactId>spring-core</artifactId> <artifactId>assertj-core</artifactId>
<version>${spring-version}</version> <version>1.5.0</version>
</dependency> <scope>test</scope>
<dependency> </dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId> <dependency>
<version>${spring-version}</version> <groupId>mysql</groupId>
</dependency> <artifactId>mysql-connector-java</artifactId>
<version>5.1.18</version>
<dependency> </dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId> <dependency>
<version>3.0.1</version> <groupId>commons-dbcp</groupId>
</dependency> <artifactId>commons-dbcp</artifactId>
<dependency> <version>1.3</version>
<groupId>org.springframework</groupId> </dependency>
<artifactId>spring-context</artifactId>
<version>${spring-version}</version> <dependency>
</dependency> <groupId>org.springframework</groupId>
<dependency> <artifactId>spring-aop</artifactId>
<groupId>org.springframework</groupId> <version>${spring-version}</version>
<artifactId>spring-context-support</artifactId> </dependency>
<version>${spring-version}</version>
</dependency> <dependency>
<dependency> <groupId>org.aspectj</groupId>
<groupId>com.alibaba</groupId> <artifactId>aspectjrt</artifactId>
<artifactId>fastjson</artifactId> <version>1.7.2</version>
<version>1.1.37</version> </dependency>
</dependency> <dependency>
<groupId>org.aspectj</groupId>
</dependencies> <artifactId>aspectjweaver</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.1.37</version>
</dependency>
</dependencies>
</dependencyManagement>
<build> <build>
<plugins> <plugins>
...@@ -104,4 +140,4 @@ ...@@ -104,4 +140,4 @@
</build> </build>
</project> </project>
\ No newline at end of file
webmagic-avalon/src/main/resources/spring/applicationContext-myBatis.xml deleted 100644 → 0
View file @ 74fdd1df
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<!--<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">-->
<!--<property name="dataSource" ref="dataSource" />-->
<!--</bean>-->
<!--<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">-->
<!--<property name="basePackage" value="us.codecraft.blackhole.suite.dao" />-->
<!--</bean>-->
<!--<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"-->
<!--destroy-method="close">-->
<!--<property name="driverClassName" value="org.sqlite.JDBC" />-->
<!--<property name="url" value="jdbc:sqlite:/usr/local/hostd/zonesfile.db" />-->
<!--</bean>-->
</beans>
\ No newline at end of file
webmagic-avalon/webmagic-admin/README.md 0 → 100644
View file @ 04ade756
WebMagic-Admin
=====
Admin is the control web of workers.
\ No newline at end of file
webmagic-panel/pom.xml → webmagic-avalon/webmagic-admin/pom.xml
View file @ 04ade756
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent> <parent>
<artifactId>webmagic-parent</artifactId> <artifactId>webmagic-avalon</artifactId>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<version>0.4.3-SNAPSHOT</version> <version>0.5.0</version>
</parent> </parent>
<modelVersion>4.0.0</modelVersion> <modelVersion>4.0.0</modelVersion>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>webmagic-panel</artifactId> <artifactId>webmagic-admin</artifactId>
<packaging>war</packaging>
<dependencies> <dependencies>
<dependency> <dependency>
<groupId>us.codecraft</groupId> <groupId>us.codecraft</groupId>
<artifactId>webmagic-scripts</artifactId> <artifactId>webmagic-avalon-common</artifactId>
<version>${project.version}</version> <version>${project.version}</version>
</dependency> </dependency>
</dependencies> </dependencies>
<build> <build>
......
webmagic-avalon/webmagic-admin/src/main/resources/log/log4j.xml 0 → 100644
View file @ 04ade756
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yy-MM-dd HH:mm:ss,SSS} %-5p %c(%F:%L) ## %m%n" />
</layout>
</appender>
<logger name="org.apache" additivity="false">
<level value="warn" />
<appender-ref ref="stdout" />
</logger>
<root>
<level value="info" />
<appender-ref ref="stdout" />
</root>
</log4j:configuration>
webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/pages/create_spider.ftl 0 → 100644
View file @ 04ade756
<html>
<head>
<script src=""></script>
</head>
<div class="url-box">
<input type="text" id="url-input">
</div>
<div class="content-show">
</div>
</html>
\ No newline at end of file
webmagic-avalon/src/main/webapp/WEB-INF/pages/dashboard.ftl → webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/pages/dashboard.ftl
View file @ 04ade756
...@@ -15,8 +15,8 @@ ...@@ -15,8 +15,8 @@
<meta charset="utf-8"> <meta charset="utf-8">
<title>WebMaigc Avalon</title> <title>WebMaigc Avalon</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Charisma, a fully featured, responsive, HTML5, Bootstrap admin template."> <meta name="description" content="WebMagic, a scalable crawler framework.">
<meta name="author" content="Muhammad Usman"> <meta name="author" content="code4crafter@gmail.com">
<!-- The styles --> <!-- The styles -->
<link id="bs-css" href="${staticPath}/css/bootstrap-cerulean.css" rel="stylesheet"> <link id="bs-css" href="${staticPath}/css/bootstrap-cerulean.css" rel="stylesheet">
...@@ -123,23 +123,10 @@ ...@@ -123,23 +123,10 @@
<div class="well nav-collapse sidebar-nav"> <div class="well nav-collapse sidebar-nav">
<ul class="nav nav-tabs nav-stacked main-menu"> <ul class="nav nav-tabs nav-stacked main-menu">
<li class="nav-header hidden-tablet">Main</li> <li class="nav-header hidden-tablet">Main</li>
<li><a class="ajax-link" href="index.html"><i class="icon-home"></i><span class="hidden-tablet"> Dashboard</span></a></li> <li><a class="ajax-link" href="dashboard"><i class="icon-home"></i><span class="hidden-tablet">Dashboard</span></a></li>
<li><a class="ajax-link" href="ui.html"><i class="icon-eye-open"></i><span class="hidden-tablet">Spider</span></a></li> <li><a class="ajax-link" href="spider"><i class="icon-eye-open"></i><span class="hidden-tablet">Spider</span></a></li>
<li><a class="ajax-link" href="form.html"><i class="icon-edit"></i><span class="hidden-tablet">Charts</span></a></li> <li><a class="ajax-link" href="charts"><i class="icon-edit"></i><span class="hidden-tablet">Charts</span></a></li>
<li><a class="ajax-link" href="chart.html"><i class="icon-list-alt"></i><span class="hidden-tablet"> Charts</span></a></li>
<li><a class="ajax-link" href="typography.html"><i class="icon-font"></i><span class="hidden-tablet"> Typography</span></a></li>
<li><a class="ajax-link" href="gallery.html"><i class="icon-picture"></i><span class="hidden-tablet"> Gallery</span></a></li>
<li class="nav-header hidden-tablet">Sample Section</li>
<li><a class="ajax-link" href="table.html"><i class="icon-align-justify"></i><span class="hidden-tablet"> Tables</span></a></li>
<li><a class="ajax-link" href="calendar.html"><i class="icon-calendar"></i><span class="hidden-tablet"> Calendar</span></a></li>
<li><a class="ajax-link" href="grid.html"><i class="icon-th"></i><span class="hidden-tablet"> Grid</span></a></li>
<li><a class="ajax-link" href="file-manager.html"><i class="icon-folder-open"></i><span class="hidden-tablet"> File Manager</span></a></li>
<li><a href="tour.html"><i class="icon-globe"></i><span class="hidden-tablet"> Tour</span></a></li>
<li><a class="ajax-link" href="icon.html"><i class="icon-star"></i><span class="hidden-tablet"> Icons</span></a></li>
<li><a href="error.html"><i class="icon-ban-circle"></i><span class="hidden-tablet"> Error Page</span></a></li>
<li><a href="login.html"><i class="icon-lock"></i><span class="hidden-tablet"> Login Page</span></a></li>
</ul> </ul>
<label id="for-is-ajax" class="hidden-tablet" for="is-ajax"><input id="is-ajax" type="checkbox"> Ajax on menu</label>
</div><!--/.well --> </div><!--/.well -->
</div><!--/span--> </div><!--/span-->
<!-- left menu ends --> <!-- left menu ends -->
...@@ -173,26 +160,6 @@ ...@@ -173,26 +160,6 @@
<span class="notification">6</span> <span class="notification">6</span>
</a> </a>
<a data-rel="tooltip" title="4 new pro members." class="well span3 top-block" href="#">
<span class="icon32 icon-color icon-star-on"></span>
<div>Pro Members</div>
<div>228</div>
<span class="notification green">4</span>
</a>
<a data-rel="tooltip" title="$34 new sales." class="well span3 top-block" href="#">
<span class="icon32 icon-color icon-cart"></span>
<div>Sales</div>
<div>$13320</div>
<span class="notification yellow">$34</span>
</a>
<a data-rel="tooltip" title="12 new messages." class="well span3 top-block" href="#">
<span class="icon32 icon-color icon-envelope-closed"></span>
<div>Messages</div>
<div>25</div>
<span class="notification red">12</span>
</a>
</div> </div>
<div class="row-fluid"> <div class="row-fluid">
......
webmagic-avalon/src/main/webapp/WEB-INF/web.xml → webmagic-avalon/webmagic-admin/src/main/webapp/WEB-INF/web.xml
View file @ 04ade756
...@@ -7,7 +7,7 @@ ...@@ -7,7 +7,7 @@
<context-param> <context-param>
<param-name>contextConfigLocation</param-name> <param-name>contextConfigLocation</param-name>
<param-value> <param-value>
classpath*:spring/applicationContext*.xml, classpath*:/config/spring/applicationContext*.xml,
</param-value> </param-value>
</context-param> </context-param>
...@@ -33,7 +33,7 @@ ...@@ -33,7 +33,7 @@
<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class> <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
<init-param> <init-param>
<param-name>contextConfigLocation</param-name> <param-name>contextConfigLocation</param-name>
<param-value>classpath:/spring/applicationContext*.xml</param-value> <param-value>classpath*:config/spring/applicationContext*.xml</param-value>
</init-param> </init-param>
<load-on-startup>1</load-on-startup> <load-on-startup>1</load-on-startup>
</servlet> </servlet>
......
webmagic-avalon/src/main/webapp/static/css/bootstrap-cerulean.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/bootstrap-cerulean.css
View file @ 04ade756
@import url(https://fonts.googleapis.com/css?family=Karla|Ubuntu); /*@import url(https://fonts.googleapis.com/css?family=Karla|Ubuntu);*/
/*! /*!
* Bootstrap v2.0.4 * Bootstrap v2.0.4
* *
......
webmagic-avalon/src/main/webapp/static/css/bootstrap-cyborg.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/bootstrap-cyborg.css
View file @ 04ade756
@import url('https://fonts.googleapis.com/css?family=Droid+Sans:400,700'); /*@import url('https://fonts.googleapis.com/css?family=Droid+Sans:400,700');*/
/*! /*!
* Bootstrap v2.0.4 * Bootstrap v2.0.4
* *
......
webmagic-avalon/src/main/webapp/static/css/bootstrap-journal.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/bootstrap-journal.css
View file @ 04ade756
@import url('https://fonts.googleapis.com/css?family=Open+Sans:400,700'); /*@import url('https://fonts.googleapis.com/css?family=Open+Sans:400,700');*/
/*! /*!
* Bootstrap v2.0.4 * Bootstrap v2.0.4
* *
......
webmagic-avalon/src/main/webapp/static/css/bootstrap-redy.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/bootstrap-redy.css
View file @ 04ade756
@import url(https://fonts.googleapis.com/css?family=Karla|Ubuntu); /*@import url(https://fonts.googleapis.com/css?family=Karla|Ubuntu);*/
/*! /*!
* Bootstrap v2.0.4 * Bootstrap v2.0.4
* *
......
webmagic-avalon/src/main/webapp/static/css/bootstrap-united.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/bootstrap-united.css
View file @ 04ade756
@import url(https://fonts.googleapis.com/css?family=Ubuntu); /*@import url(https://fonts.googleapis.com/css?family=Ubuntu);*/
/*! /*!
* Bootstrap v2.0.4 * Bootstrap v2.0.4
* *
......
webmagic-avalon/src/main/webapp/static/css/charisma-app.css → webmagic-avalon/webmagic-admin/src/main/webapp/static/css/charisma-app.css
View file @ 04ade756
@import url(https://fonts.googleapis.com/css?family=Shojumaru); /*@import url(https://fonts.googleapis.com/css?family=Shojumaru);*/
select{ select{
background-color:#fff; background-color:#fff;
......
webmagic-avalon/webmagic-avalon-common/pom.xml 0 → 100644
View file @ 04ade756
This diff is collapsed.
webmagic-avalon/webmagic-worker/README.md 0 → 100644
View file @ 04ade756
This diff is collapsed.
webmagic-core/pom.xml
View file @ 04ade756
This diff is collapsed.
webmagic-core/src/test/resources/log4j.xml
View file @ 04ade756
This diff is collapsed.
webmagic-extension/pom.xml
View file @ 04ade756
This diff is collapsed.
webmagic-extension/src/main/resouces/log4j.xml 0 → 100644
View file @ 04ade756
This diff is collapsed.
webmagic-extension/src/main/resouces/spider-config-draft.xml 0 → 100644
View file @ 04ade756
This diff is collapsed.
webmagic-lucene/README.md deleted 100644 → 0
View file @ 74fdd1df
This diff is collapsed.
webmagic-panel/README.md deleted 100644 → 0
View file @ 74fdd1df
This diff is collapsed.
webmagic-samples/pom.xml
View file @ 04ade756
This diff is collapsed.
webmagic-saxon/pom.xml
View file @ 04ade756
This diff is collapsed.
webmagic-scripts/README.md 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/deploy.sh 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/pom.xml 100644 → 100755
View file @ 04ade756
This diff is collapsed.
webmagic-scripts/src/main/resources/js/defines.js 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/src/main/resources/js/github.js 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/src/main/resources/js/oschina.js 100644 → 100755
View file @ 04ade756
This diff is collapsed.
webmagic-scripts/src/main/resources/log4j.xml 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/src/main/resources/python/defines.py 0 → 100755
View file @ 04ade756
This diff is collapsed.
webmagic-scripts/src/main/resources/python/oschina.py 0 → 100755
View file @ 04ade756
This diff is collapsed.
webmagic-scripts/src/main/resources/ruby/defines.rb 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/src/main/resources/ruby/github.rb 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-scripts/src/test/resouces/log4j.xml 100644 → 100755
View file @ 04ade756
File mode changed from 100644 to 100755
webmagic-selenium/pom.xml
View file @ 04ade756
This diff is collapsed.
zh_docs/README.md
View file @ 04ade756
This diff is collapsed.
zh_docs/user-manual-new.md 0 → 100644
View file @ 04ade756
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment