Compare commits
5 commits
393d02e925
...
dd71c4850f
Author | SHA1 | Date | |
---|---|---|---|
dd71c4850f | |||
c5a9e63682 | |||
a3315ce669 | |||
fd832f5934 | |||
960e6e6ec5 |
16 changed files with 249 additions and 48 deletions
|
@ -1,7 +1,7 @@
|
|||
PATH
|
||||
remote: .
|
||||
specs:
|
||||
webgalien (0.1.0)
|
||||
webgalen (0.1.0)
|
||||
celluloid (~> 0.17.3)
|
||||
colorize (~> 0.8.1)
|
||||
mechanize (~> 2.7)
|
||||
|
@ -82,7 +82,7 @@ DEPENDENCIES
|
|||
minitest
|
||||
rake
|
||||
rasem (~> 0.7.1)
|
||||
webgalien!
|
||||
webgalen!
|
||||
|
||||
BUNDLED WITH
|
||||
1.16.0
|
||||
|
|
165
LICENSE.txt
Normal file
165
LICENSE.txt
Normal file
|
@ -0,0 +1,165 @@
|
|||
GNU LESSER GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
|
||||
This version of the GNU Lesser General Public License incorporates
|
||||
the terms and conditions of version 3 of the GNU General Public
|
||||
License, supplemented by the additional permissions listed below.
|
||||
|
||||
0. Additional Definitions.
|
||||
|
||||
As used herein, "this License" refers to version 3 of the GNU Lesser
|
||||
General Public License, and the "GNU GPL" refers to version 3 of the GNU
|
||||
General Public License.
|
||||
|
||||
"The Library" refers to a covered work governed by this License,
|
||||
other than an Application or a Combined Work as defined below.
|
||||
|
||||
An "Application" is any work that makes use of an interface provided
|
||||
by the Library, but which is not otherwise based on the Library.
|
||||
Defining a subclass of a class defined by the Library is deemed a mode
|
||||
of using an interface provided by the Library.
|
||||
|
||||
A "Combined Work" is a work produced by combining or linking an
|
||||
Application with the Library. The particular version of the Library
|
||||
with which the Combined Work was made is also called the "Linked
|
||||
Version".
|
||||
|
||||
The "Minimal Corresponding Source" for a Combined Work means the
|
||||
Corresponding Source for the Combined Work, excluding any source code
|
||||
for portions of the Combined Work that, considered in isolation, are
|
||||
based on the Application, and not on the Linked Version.
|
||||
|
||||
The "Corresponding Application Code" for a Combined Work means the
|
||||
object code and/or source code for the Application, including any data
|
||||
and utility programs needed for reproducing the Combined Work from the
|
||||
Application, but excluding the System Libraries of the Combined Work.
|
||||
|
||||
1. Exception to Section 3 of the GNU GPL.
|
||||
|
||||
You may convey a covered work under sections 3 and 4 of this License
|
||||
without being bound by section 3 of the GNU GPL.
|
||||
|
||||
2. Conveying Modified Versions.
|
||||
|
||||
If you modify a copy of the Library, and, in your modifications, a
|
||||
facility refers to a function or data to be supplied by an Application
|
||||
that uses the facility (other than as an argument passed when the
|
||||
facility is invoked), then you may convey a copy of the modified
|
||||
version:
|
||||
|
||||
a) under this License, provided that you make a good faith effort to
|
||||
ensure that, in the event an Application does not supply the
|
||||
function or data, the facility still operates, and performs
|
||||
whatever part of its purpose remains meaningful, or
|
||||
|
||||
b) under the GNU GPL, with none of the additional permissions of
|
||||
this License applicable to that copy.
|
||||
|
||||
3. Object Code Incorporating Material from Library Header Files.
|
||||
|
||||
The object code form of an Application may incorporate material from
|
||||
a header file that is part of the Library. You may convey such object
|
||||
code under terms of your choice, provided that, if the incorporated
|
||||
material is not limited to numerical parameters, data structure
|
||||
layouts and accessors, or small macros, inline functions and templates
|
||||
(ten or fewer lines in length), you do both of the following:
|
||||
|
||||
a) Give prominent notice with each copy of the object code that the
|
||||
Library is used in it and that the Library and its use are
|
||||
covered by this License.
|
||||
|
||||
b) Accompany the object code with a copy of the GNU GPL and this license
|
||||
document.
|
||||
|
||||
4. Combined Works.
|
||||
|
||||
You may convey a Combined Work under terms of your choice that,
|
||||
taken together, effectively do not restrict modification of the
|
||||
portions of the Library contained in the Combined Work and reverse
|
||||
engineering for debugging such modifications, if you also do each of
|
||||
the following:
|
||||
|
||||
a) Give prominent notice with each copy of the Combined Work that
|
||||
the Library is used in it and that the Library and its use are
|
||||
covered by this License.
|
||||
|
||||
b) Accompany the Combined Work with a copy of the GNU GPL and this license
|
||||
document.
|
||||
|
||||
c) For a Combined Work that displays copyright notices during
|
||||
execution, include the copyright notice for the Library among
|
||||
these notices, as well as a reference directing the user to the
|
||||
copies of the GNU GPL and this license document.
|
||||
|
||||
d) Do one of the following:
|
||||
|
||||
0) Convey the Minimal Corresponding Source under the terms of this
|
||||
License, and the Corresponding Application Code in a form
|
||||
suitable for, and under terms that permit, the user to
|
||||
recombine or relink the Application with a modified version of
|
||||
the Linked Version to produce a modified Combined Work, in the
|
||||
manner specified by section 6 of the GNU GPL for conveying
|
||||
Corresponding Source.
|
||||
|
||||
1) Use a suitable shared library mechanism for linking with the
|
||||
Library. A suitable mechanism is one that (a) uses at run time
|
||||
a copy of the Library already present on the user's computer
|
||||
system, and (b) will operate properly with a modified version
|
||||
of the Library that is interface-compatible with the Linked
|
||||
Version.
|
||||
|
||||
e) Provide Installation Information, but only if you would otherwise
|
||||
be required to provide such information under section 6 of the
|
||||
GNU GPL, and only to the extent that such information is
|
||||
necessary to install and execute a modified version of the
|
||||
Combined Work produced by recombining or relinking the
|
||||
Application with a modified version of the Linked Version. (If
|
||||
you use option 4d0, the Installation Information must accompany
|
||||
the Minimal Corresponding Source and Corresponding Application
|
||||
Code. If you use option 4d1, you must provide the Installation
|
||||
Information in the manner specified by section 6 of the GNU GPL
|
||||
for conveying Corresponding Source.)
|
||||
|
||||
5. Combined Libraries.
|
||||
|
||||
You may place library facilities that are a work based on the
|
||||
Library side by side in a single library together with other library
|
||||
facilities that are not Applications and are not covered by this
|
||||
License, and convey such a combined library under terms of your
|
||||
choice, if you do both of the following:
|
||||
|
||||
a) Accompany the combined library with a copy of the same work based
|
||||
on the Library, uncombined with any other library facilities,
|
||||
conveyed under the terms of this License.
|
||||
|
||||
b) Give prominent notice with the combined library that part of it
|
||||
is a work based on the Library, and explaining where to find the
|
||||
accompanying uncombined form of the same work.
|
||||
|
||||
6. Revised Versions of the GNU Lesser General Public License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions
|
||||
of the GNU Lesser General Public License from time to time. Such new
|
||||
versions will be similar in spirit to the present version, but may
|
||||
differ in detail to address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Library as you received it specifies that a certain numbered version
|
||||
of the GNU Lesser General Public License "or any later version"
|
||||
applies to it, you have the option of following the terms and
|
||||
conditions either of that published version or of any later version
|
||||
published by the Free Software Foundation. If the Library as you
|
||||
received it does not specify a version number of the GNU Lesser
|
||||
General Public License, you may choose any version of the GNU Lesser
|
||||
General Public License ever published by the Free Software Foundation.
|
||||
|
||||
If the Library as you received it specifies that a proxy can decide
|
||||
whether future versions of the GNU Lesser General Public License shall
|
||||
apply, that proxy's public statement of acceptance of any version is
|
||||
permanent authorization for you to choose that version for the
|
||||
Library.
|
46
README.md
46
README.md
|
@ -1,13 +1,49 @@
|
|||
# Webgalen
|
||||
|
||||
Prerequisites
|
||||
A tool to dissect of websites (tribute to [Galen of Pergamon](https://en.wikipedia.org/wiki/Galen))
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install a few tools
|
||||
|
||||
apt-get install img2pdf imagemagick
|
||||
|
||||
|
||||
Increase memory in imagemagick policy `/etc/ImageMagick-6/policy.xml`
|
||||
|
||||
<policy domain="resource" name="memory" value="2GiB"/>
|
||||
<policy domain="resource" name="memory" value="2GiB"/>
|
||||
|
||||
window.$FlowPaper('documentViewer').config
|
||||
window.$FlowPaper('documentViewer').switchMode('portrait')
|
||||
## Usage
|
||||
|
||||
$ be exe/webgalen
|
||||
Commands:
|
||||
webgalen help [COMMAND] # Describe available commands or one specific command
|
||||
webgalen list-devices # list available profiles
|
||||
webgalen screenshot FILE # take screenshots for each page
|
||||
webgalen sitemap URL FILE # crawl site and export sitemap
|
||||
|
||||
Options:
|
||||
-ua, [--user-agent=USER-AGENT] # choose user agent (default Mozilla)
|
||||
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork it ( http://github.com/glenux/webgalen/fork )
|
||||
2. Create your feature branch (`git checkout -b my-new-feature`)
|
||||
3. Commit your changes (`git commit -am 'Add some feature'`)
|
||||
4. Push to the branch (`git push origin my-new-feature`)
|
||||
5. Create new Pull Request
|
||||
|
||||
|
||||
## Credits
|
||||
|
||||
Author & Maintainer: Glenn Y. ROLLAND
|
||||
|
||||
Contributors: none yet ;)
|
||||
|
||||
Got questions? Need help? Tweet at @glenux
|
||||
|
||||
|
||||
## License
|
||||
|
||||
Webgalen is Copyright © 2018 Glenn ROLLAND. It is free software, and may be redistributed under the terms specified in the LICENSE file.
|
||||
|
||||
|
|
6
exe/webgalen
Executable file
6
exe/webgalen
Executable file
|
@ -0,0 +1,6 @@
|
|||
#!/usr/bin/env ruby
|
||||
|
||||
require 'webgalen'
|
||||
|
||||
Webgalen::Cli.start(ARGV)
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
#!/usr/bin/env ruby
|
||||
|
||||
require 'webgalien'
|
||||
|
||||
Webgalien::Cli.start(ARGV)
|
||||
|
23
lib/webgalen.rb
Normal file
23
lib/webgalen.rb
Normal file
|
@ -0,0 +1,23 @@
|
|||
|
||||
# Load external dependencies
|
||||
require 'thor'
|
||||
require 'celluloid/current'
|
||||
require 'mechanize'
|
||||
require 'selenium-webdriver'
|
||||
require 'yaml'
|
||||
require 'thor'
|
||||
require 'colorize'
|
||||
require 'fileutils'
|
||||
|
||||
# Load actors
|
||||
require 'webgalen/actors/crop_png_actor'
|
||||
require 'webgalen/actors/screenshot_actor'
|
||||
require 'webgalen/actors/work_actor'
|
||||
|
||||
require 'webgalen/screenshot'
|
||||
require 'webgalen/sitemap'
|
||||
require 'webgalen/devices'
|
||||
|
||||
# Load cli
|
||||
require 'webgalen/cli'
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
|
||||
# Actor for cropping png
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class CropPngActor
|
||||
include Celluloid
|
||||
|
|
@ -4,7 +4,7 @@ USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) ' \
|
|||
# configure the driver to run in headless mode
|
||||
# Selenium::WebDriver.logger.level =
|
||||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class ScreenshotActor
|
||||
include Celluloid
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class Work
|
||||
attr_reader :id, :input, :artefacts
|
||||
attr_accessor :output
|
|
@ -1,7 +1,7 @@
|
|||
|
||||
require 'table_print'
|
||||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class Cli < Thor
|
||||
class_option :'user-agent',
|
||||
aliases: '-ua',
|
|
@ -1,6 +1,6 @@
|
|||
require 'csv'
|
||||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class Devices
|
||||
PROFILES_CSV = <<-MARK
|
||||
laptop, 1440, 900, computer, "x"
|
|
@ -1,5 +1,5 @@
|
|||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class Screenshot
|
||||
def self.start(sitemap:, output_path: , device:, orientation:)
|
||||
config = YAML.load File.open(sitemap)
|
|
@ -1,5 +1,5 @@
|
|||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
class Sitemap
|
||||
def self.start(url:, output:)
|
||||
puts "Loading #{url}"
|
|
@ -1,4 +1,4 @@
|
|||
|
||||
module Webgalien
|
||||
module Webgalen
|
||||
VERSION = '0.1.0'.freeze
|
||||
end
|
|
@ -1,23 +0,0 @@
|
|||
|
||||
# Load external dependencies
|
||||
require 'thor'
|
||||
require 'celluloid/current'
|
||||
require 'mechanize'
|
||||
require 'selenium-webdriver'
|
||||
require 'yaml'
|
||||
require 'thor'
|
||||
require 'colorize'
|
||||
require 'fileutils'
|
||||
|
||||
# Load actors
|
||||
require 'webgalien/actors/crop_png_actor'
|
||||
require 'webgalien/actors/screenshot_actor'
|
||||
require 'webgalien/actors/work_actor'
|
||||
|
||||
require 'webgalien/screenshot'
|
||||
require 'webgalien/sitemap'
|
||||
require 'webgalien/devices'
|
||||
|
||||
# Load cli
|
||||
require 'webgalien/cli'
|
||||
|
|
@ -1,16 +1,16 @@
|
|||
# coding: utf-8
|
||||
lib = File.expand_path('../lib', __FILE__)
|
||||
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
||||
require 'webgalien/version'
|
||||
require 'webgalen/version'
|
||||
|
||||
Gem::Specification.new do |spec|
|
||||
spec.name = "webgalien"
|
||||
spec.version = Webgalien::VERSION
|
||||
spec.name = "webgalen"
|
||||
spec.version = Webgalen::VERSION
|
||||
spec.authors = ["Glenn Y. Rolland"]
|
||||
spec.email = ["glenux@glenux.net"]
|
||||
spec.summary = %q{Use GIT logs to give an estimation of spent time & costs of your projects.}
|
||||
spec.description = %q{Use GIT logs to give an estimation of spent time & costs of your projects.}
|
||||
spec.homepage = "https://github.com/glenux/webgalien"
|
||||
spec.homepage = "https://github.com/glenux/webgalen"
|
||||
spec.license = "MIT"
|
||||
|
||||
spec.files = `git ls-files -z`.split("\x0")
|
Loading…
Add table
Reference in a new issue